2021-01-22T09:58:34.906 INFO:root:teuthology version: 1.0.0-fe79ba1b
2021-01-22T09:58:34.908 INFO:teuthology.results:seed: '9087'
2021-01-22T11:22:23.606 INFO:root:teuthology version: 1.0.0-fe79ba1b
2021-01-22T11:22:23.656 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-01-22T23:02:38.765 INFO:teuthology.results:Tests finished! gathering results...
2021-01-22T23:02:51.803 INFO:scrape:Found 100 jobs
2021-01-22T23:03:02.627 INFO:scrape:Found 9 distinct failure reasons
2021-01-22T23:03:02.628 INFO:scrape:Failure: "2021-01-22T11:07:13.997961+0000 mds.f (mds.0) 14 : cluster [WRN] Scrub error on inode 0x100000001fe (/client.0/tmp/blogbench-1.0/src) see mds.f log and `damage ls` output for details" in cluster log
2021-01-22T23:03:02.629 INFO:scrape:4 jobs: ['5815256', '5815278', '5815228', '5815249']
2021-01-22T23:03:02.629 INFO:scrape:suites intersection: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-22T23:03:02.630 INFO:scrape:suites union: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/workload/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'ranks/3', 'ranks/5', 'rhel_8}', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}', 'workunit/suites/blogbench}}', 'workunit/suites/pjd}}']
2021-01-22T23:03:02.630 INFO:scrape:
2021-01-22T23:03:02.631 INFO:scrape:Dead: b'2021-01-22T23:00:17.464 DEBUG:teuthology.exit:Finished running handlers'
2021-01-22T23:03:02.631 INFO:scrape:1 jobs: ['5815275']
2021-01-22T23:03:02.632 INFO:scrape:suites: ['2-workunit/cfuse_workunit_suites_fsstress}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/multifs/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-22T23:03:02.632 INFO:scrape:
2021-01-22T23:03:02.632 INFO:scrape:Failure: "2021-01-22T10:52:02.780642+0000 mds.f (mds.0) 19 : cluster [WRN] Scrub error on inode 0x100000003b1 (/client.0/tmp/fsstress/ltp-full-20091231/testcases/commands) see mds.f log and `damage ls` output for details" in cluster log
2021-01-22T23:03:02.633 INFO:scrape:1 jobs: ['5815238']
2021-01-22T23:03:02.633 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/3', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/fsstress}}']
2021-01-22T23:03:02.634 INFO:scrape:
2021-01-22T23:03:02.635 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi086 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=43852f5d7b2e222d68cc5fc492ef8552d858678d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'
2021-01-22T23:03:02.635 INFO:scrape:1 jobs: ['5815292']
2021-01-22T23:03:02.636 INFO:scrape:suites: ['2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-22T23:03:02.637 INFO:scrape:
2021-01-22T23:03:02.637 INFO:scrape:Crash: Command failed on smithi133 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp'
2021-01-22T23:03:02.638 INFO:scrape:ceph version 16.0.0-9215-g43852f5d (43852f5d7b2e222d68cc5fc492ef8552d858678d) pacific (dev)
 1: /lib64/libpthread.so.0(+0x12dc0) [0xf9bddc0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x50b9a23]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x27bbec) [0x50b9bec]
 6: (Monitor::~Monitor()+0xc2d) [0x3e879d]
 7: (Monitor::~Monitor()+0xd) [0x3e87fd]
 8: main()
 9: __libc_start_main()
 10: _start()
2021-01-22T23:03:02.638 INFO:scrape:1 jobs: ['5815276']
2021-01-22T23:03:02.639 INFO:scrape:suites: ['centos_latest', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/verify/{begin', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag_enable', 'ranks/3', 'session_timeout', 'tasks/fsstress', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-22T23:03:02.639 INFO:scrape:
2021-01-22T23:03:02.639 INFO:scrape:Failure: "2021-01-22T11:17:13.712809+0000 mon.b (mon.0) 256 : cluster [WRN] Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)" in cluster log
2021-01-22T23:03:02.640 INFO:scrape:1 jobs: ['5815259']
2021-01-22T23:03:02.640 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/3', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/ffsb}}']
2021-01-22T23:03:02.641 INFO:scrape:
2021-01-22T23:03:02.641 INFO:scrape:Failure: "2021-01-22T11:42:33.184029+0000 mds.d (mds.0) 14 : cluster [WRN] Scrub error on inode 0x10000000c24 (/client.0/tmp/fsstress/ltp-full-20091231/testcases/open_hpi_testsuite) see mds.d log and `damage ls` output for details" in cluster log
2021-01-22T23:03:02.641 INFO:scrape:1 jobs: ['5815289']
2021-01-22T23:03:02.642 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/5', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/fsstress}}']
2021-01-22T23:03:02.643 INFO:scrape:
2021-01-22T23:03:02.643 INFO:scrape:Failure: "2021-01-22T11:47:15.773625+0000 mds.d (mds.2) 4 : cluster [WRN] Scrub error on inode 0x10000000265 (/client.0/tmp/testdir/dir1/dir2/dir3) see mds.d log and `damage ls` output for details" in cluster log
2021-01-22T23:03:02.644 INFO:scrape:1 jobs: ['5815304']
2021-01-22T23:03:02.644 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{rhel_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/3', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}']
2021-01-22T23:03:02.645 INFO:scrape:
2021-01-22T23:03:02.645 INFO:scrape:Crash: Command failed (workunit test suites/blogbench.sh) on smithi001 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=43852f5d7b2e222d68cc5fc492ef8552d858678d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh'
2021-01-22T23:03:02.646 INFO:scrape:ceph version 16.0.0-9215-g43852f5d (43852f5d7b2e222d68cc5fc492ef8552d858678d) pacific (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fe7e11ac980]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7fe7e233163e]
 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fe7e23317c8]
 6: ceph-mon(+0x7a2fd2) [0x55e811fa2fd2]
 7: (Monitor::~Monitor()+0x9) [0x55e811fa3019]
 8: main()
 9: __libc_start_main()
 10: _start()
2021-01-22T23:03:02.646 INFO:scrape:1 jobs: ['5815282']
2021-01-22T23:03:02.647 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/3', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/blogbench}}']
2021-01-22T23:03:02.647 INFO:scrape:
2021-01-22T23:03:02.654 INFO:teuthology.results:Sending results to jcollin@redhat.com: Test Run: jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/
failed:  11
dead:    1
running: 0
waiting: 0
queued:  0
passed:  88


Fail
=================================================================
[5815228]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:37:59
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815228/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815228/

    "2021-01-22T10:22:33.898399+0000 mds.f (mds.0) 19 : cluster [WRN] Scrub
    error on inode 0x10000000263 (/client.0/tmp/testdir/dir1) see mds.f log and
    `damage ls` output for details" in cluster log

[5815238]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:15:45
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815238/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815238/

    "2021-01-22T10:52:02.780642+0000 mds.f (mds.0) 19 : cluster [WRN] Scrub
    error on inode 0x100000003b1 (/client.0/tmp/fsstress/ltp-
    full-20091231/testcases/commands) see mds.f log and `damage ls` output for
    details" in cluster log

[5815249]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:13:07
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815249/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815249/

    "2021-01-22T11:02:58.963127+0000 mds.f (mds.0) 14 : cluster [WRN] Scrub
    error on inode 0x100000002cd (/client.0/tmp/tmp) see mds.f log and `damage
    ls` output for details" in cluster log

[5815256]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:32:34
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815256/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815256/

    "2021-01-22T11:07:13.997961+0000 mds.f (mds.0) 14 : cluster [WRN] Scrub
    error on inode 0x100000001fe (/client.0/tmp/blogbench-1.0/src) see mds.f
    log and `damage ls` output for details" in cluster log

[5815259]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:52:58
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815259/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815259/

    "2021-01-22T11:17:13.712809+0000 mon.b (mon.0) 256 : cluster [WRN] Health
    check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)" in
    cluster log

[5815276]  fs/verify/{begin centos_latest clusters/1a5s-mds-1c-client conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   01:32:08
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815276/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815276/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0d20d3f6e20349da80a8cfc06c4d7086

    Command failed on smithi133 with status 1: 'sudo rm -rf --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp'

[5815278]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:37:34
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815278/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815278/

    "2021-01-22T11:52:04.421807+0000 mds.d (mds.1) 2 : cluster [WRN] Scrub
    error on inode 0x10000000264 (/client.0/tmp/testdir/dir1/dir2) see mds.d
    log and `damage ls` output for details" in cluster log

[5815282]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:51:19
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815282/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815282/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9be03e45e486405ba4b1543fc1ebea09

    Command failed (workunit test suites/blogbench.sh) on smithi001 with status
    135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=43852f5d7b2e222d68cc5fc492ef8552d858678d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh'

[5815289]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:14:23
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815289/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815289/

    "2021-01-22T11:42:33.184029+0000 mds.d (mds.0) 14 : cluster [WRN] Scrub
    error on inode 0x10000000c24 (/client.0/tmp/fsstress/ltp-
    full-20091231/testcases/open_hpi_testsuite) see mds.d log and `damage ls`
    output for details" in cluster log

[5815292]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:13:37
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815292/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815292/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f13b6508da02443ba05c646695630534

    Command failed (workunit test suites/pjd.sh) on smithi086 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=43852f5d7b2e222d68cc5fc492ef8552d858678d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[5815304]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:37:31
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815304/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815304/

    "2021-01-22T11:47:15.773625+0000 mds.d (mds.2) 4 : cluster [WRN] Scrub
    error on inode 0x10000000265 (/client.0/tmp/testdir/dir1/dir2/dir3) see
    mds.d log and `damage ls` output for details" in cluster log



Dead
=================================================================
[5815275]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815275/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815275/



Pass
=================================================================
[5815210] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:12:32
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815210/

[5815211] fs/bugs/client_trim_caps/{begin clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/trim-i22073}
time:   00:07:31
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815211/

[5815212] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls}
time:   00:10:18
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815212/

[5815213] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client}
time:   00:07:29
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815213/

[5815214] fs/mirror/{begin cephfs-mirror/{one-per-cluster} clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{centos_8} tasks/mirror}
time:   00:48:19
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815214/

[5815215] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{frag_enable osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:17:14
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815215/

[5815216] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   00:20:30
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815216/

[5815217] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover}
time:   00:39:07
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815217/

[5815218] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:11:16
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815218/

[5815219] fs/shell/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cephfs-shell}
time:   00:32:05
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815219/

[5815220] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:37:50
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815220/

[5815221] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
time:   03:19:39
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815221/

[5815222] fs/top/{begin cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/whitelist_health supported-random-distros$/{centos_8} tasks/fstop}
time:   00:09:33
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815222/

[5815223] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:19:35
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815223/

[5815224] fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}}
time:   00:30:37
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815224/

[5815225] fs/verify/{begin centos_latest clusters/1a5s-mds-1c-client conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/dbench validater/lockdep}
time:   00:56:35
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815225/

[5815226] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}}
time:   00:44:32
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815226/

[5815227] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/direct_io}}
time:   00:11:57
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815227/

[5815229] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:08:56
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815229/

[5815230] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}}
time:   00:39:12
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815230/

[5815231] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:35:46
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815231/

[5815232] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}}
time:   00:18:51
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815232/

[5815233] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/dbench}}
time:   00:43:51
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815233/

[5815234] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:20:13
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815234/

[5815235] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}}
time:   00:55:06
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815235/

[5815236] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:09:24
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815236/

[5815237] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/norstats}}
time:   00:15:45
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815237/

[5815239] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:12:01
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815239/

[5815240] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/fsx}}
time:   00:43:47
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815240/

[5815241] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:12:13
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815241/

[5815242] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}}
time:   00:11:14
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815242/

[5815243] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}}
time:   00:23:37
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815243/

[5815244] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   01:23:18
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815244/

[5815245] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin}
time:   00:24:15
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815245/

[5815246] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/iogen}}
time:   00:52:24
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815246/

[5815247] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:38:02
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815247/

[5815248] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/iozone}}
time:   00:12:08
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815248/

[5815250] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:14:51
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815250/

[5815251] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/direct_io}}
time:   00:14:16
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815251/

[5815252] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:10:04
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815252/

[5815253] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/fs/misc}}
time:   00:46:42
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815253/

[5815254] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}}
time:   01:58:44
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815254/

[5815255] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:20:28
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815255/

[5815257] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:13:46
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815257/

[5815258] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/dbench}}
time:   00:52:15
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815258/

[5815260] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:03:37
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815260/

[5815261] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/norstats}}
time:   00:13:39
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815261/

[5815262] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:34:08
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815262/

[5815263] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool}
time:   00:14:08
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815263/

[5815264] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}}
time:   00:15:02
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815264/

[5815265] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/fsx}}
time:   00:24:56
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815265/

[5815266] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:18:09
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815266/

[5815267] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}}
time:   00:11:46
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815267/

[5815268] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:09:56
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815268/

[5815269] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}}
time:   00:18:06
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815269/

[5815270] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/iogen}}
time:   00:55:49
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815270/

[5815271] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:12:53
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815271/

[5815272] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/iozone}}
time:   00:18:37
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815272/

[5815273] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:12:13
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815273/

[5815274] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/pjd}}
time:   00:13:02
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815274/

[5815277] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/direct_io}}
time:   00:13:15
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815277/

[5815279] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:05:49
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815279/

[5815280] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}}
time:   00:55:50
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815280/

[5815281] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:36:49
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815281/

[5815283] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/asok_dump_tree}
time:   00:20:44
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815283/

[5815284] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/dbench}}
time:   01:10:03
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815284/

[5815285] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:15:41
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815285/

[5815286] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}}
time:   00:26:43
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815286/

[5815287] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:09:21
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815287/

[5815288] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/norstats}}
time:   00:17:13
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815288/

[5815290] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:19:30
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815290/

[5815291] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/fsx}}
time:   00:26:56
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815291/

[5815293] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}}
time:   00:10:36
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815293/

[5815294] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}}
time:   00:23:30
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815294/

[5815295] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   02:16:53
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815295/

[5815296] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/iogen}}
time:   00:31:21
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815296/

[5815297] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:40:18
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815297/

[5815298] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/iozone}}
time:   00:12:33
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815298/

[5815299] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/pjd}}
time:   00:11:34
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815299/

[5815300] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:19:48
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815300/

[5815301] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair}
time:   00:10:09
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815301/

[5815302] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/direct_io}}
time:   00:15:08
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815302/

[5815303] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:10:20
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815303/

[5815305] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}}
time:   02:23:00
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815305/

[5815306] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:10:31
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815306/

[5815307] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}}
time:   00:21:21
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815307/

[5815308] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:11:52
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815308/

[5815309] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/dbench}}
time:   01:09:41
info:   http://pulpito.ceph.com/jcollin-2021-01-22_09:57:40-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5815309/