2021-01-27T09:13:22.173 INFO:root:teuthology version: 1.0.0-fe79ba1b
2021-01-27T09:13:22.175 INFO:teuthology.results:seed: '7798'
2021-01-27T10:12:20.598 INFO:root:teuthology version: 1.0.0-fe79ba1b
2021-01-27T10:12:20.633 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-01-27T21:54:36.139 INFO:teuthology.results:Tests finished! gathering results...
2021-01-27T21:54:52.804 INFO:scrape:Found 100 jobs
2021-01-27T21:55:14.963 INFO:scrape:Found 8 distinct failure reasons
2021-01-27T21:55:14.967 INFO:scrape:Failure: "2021-01-27T09:40:52.361223+0000 mds.c (mds.0) 39 : cluster [WRN] Scrub error on inode 0x10000007a73 (/client.0/tmp/payload.1/multiple_rsync_payload.146341/firmware) see mds.c log and `damage ls` output for details" in cluster log
2021-01-27T21:55:14.968 INFO:scrape:6 jobs: ['5833385', '5833413', '5833446', '5833435', '5833406', '5833395']
2021-01-27T21:55:14.968 INFO:scrape:suites intersection: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag_enable', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-27T21:55:14.969 INFO:scrape:suites union: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/3', 'ranks/5', 'rhel_8}', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}', 'workunit/suites/blogbench}}', 'workunit/suites/fsstress}}', 'workunit/suites/pjd}}']
2021-01-27T21:55:14.969 INFO:scrape:
2021-01-27T21:55:14.970 INFO:scrape:Dead: b'2021-01-27T21:29:39.122 DEBUG:teuthology.exit:Finished running handlers'
2021-01-27T21:55:14.971 INFO:scrape:ceph version 17.0.0-139-g164b0f63 (164b0f632a5fe3384fa7cbb10929c65ed7cc0f12) pacific (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f6adb6db3c0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1ad) [0x7f6adb987d8a]
 5: /usr/lib/ceph/libceph-common.so.2(+0x263f37) [0x7f6adb987f37]
 6: (MonClient::shutdown()+0x911) [0x7f6adbd5cb31]
 7: (MonClient::get_monmap_and_config()+0x4c8) [0x7f6adbd5f608]
 8: (global_init(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const*, std::vector<char const*, std::allocator<char const*> >&, unsigned int, code_environment_t, int, bool)+0x190f) [0x55724403439f]
 9: main()
 10: __libc_start_main()
 11: _start()
2021-01-27T21:55:14.971 INFO:scrape:1 jobs: ['5833403']
2021-01-27T21:55:14.971 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{rhel_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/5', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/iogen}}']
2021-01-27T21:55:14.972 INFO:scrape:
2021-01-27T21:55:14.972 INFO:scrape:Failure: SSH connection to smithi080 was lost: 'sudo rm /etc/logrotate.d/ceph-test.conf'
2021-01-27T21:55:14.973 INFO:scrape:5 jobs: ['5833440', '5833438', '5833453', '5833410', '5833416']
2021-01-27T21:55:14.973 INFO:scrape:suites intersection: ['conf/{client', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-27T21:55:14.974 INFO:scrape:suites union: ['2-workunit/suites/ffsb}}', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'fs/thrash/workloads/{begin', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'no_client_pidfile', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/1', 'ranks/3', 'scrub/yes', 'session_timeout', 'tasks/asok_dump_tree}', 'tasks/{0-check-counter', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}', 'workunit/suites/ffsb}}', 'workunit/suites/iogen}}']
2021-01-27T21:55:14.974 INFO:scrape:
2021-01-27T21:55:14.974 INFO:scrape:Failure: Unable to open channel.
2021-01-27T21:55:14.975 INFO:scrape:1 jobs: ['5833461']
2021-01-27T21:55:14.975 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{rhel_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/3', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}']
2021-01-27T21:55:14.976 INFO:scrape:
2021-01-27T21:55:14.976 INFO:scrape:Dead: b'2021-01-27T21:51:59.537 DEBUG:teuthology.exit:Finished running handlers'
2021-01-27T21:55:14.976 INFO:scrape:3 jobs: ['5833439', '5833432', '5833404']
2021-01-27T21:55:14.977 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{frag_enable', 'session_timeout', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-27T21:55:14.977 INFO:scrape:suites union: ['2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/suites/ffsb}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin', 'fs/thrash/workloads/{begin', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/3', 'rhel_8}', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/blogbench}}']
2021-01-27T21:55:14.978 INFO:scrape:
2021-01-27T21:55:14.978 INFO:scrape:Crash: Command failed (workunit test suites/fsstress.sh) on smithi154 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=164b0f632a5fe3384fa7cbb10929c65ed7cc0f12 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'
2021-01-27T21:55:14.979 INFO:scrape:ceph version 17.0.0-139-g164b0f63 (164b0f632a5fe3384fa7cbb10929c65ed7cc0f12) pacific (dev)
 1: /lib64/libpthread.so.0(+0x12dc0) [0xf9bddc0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x50b9a23]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x27bbec) [0x50b9bec]
 6: (Monitor::~Monitor()+0xc2d) [0x3e889d]
 7: (Monitor::~Monitor()+0xd) [0x3e88fd]
 8: main()
 9: __libc_start_main()
 10: _start()
2021-01-27T21:55:14.979 INFO:scrape:1 jobs: ['5833433']
2021-01-27T21:55:14.980 INFO:scrape:suites: ['centos_latest', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/verify/{begin', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag_enable', 'ranks/3', 'session_timeout', 'tasks/fsstress', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-27T21:55:14.980 INFO:scrape:
2021-01-27T21:55:14.981 INFO:scrape:Failure: [Errno 113] No route to host
2021-01-27T21:55:14.981 INFO:scrape:1 jobs: ['5833389']
2021-01-27T21:55:14.982 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{rhel_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/1', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/blogbench}}']
2021-01-27T21:55:14.982 INFO:scrape:
2021-01-27T21:55:14.983 INFO:scrape:Failure: Command failed on smithi031 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master'
2021-01-27T21:55:14.983 INFO:scrape:1 jobs: ['5833414']
2021-01-27T21:55:14.984 INFO:scrape:suites: ['2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-27T21:55:14.984 INFO:scrape:
2021-01-27T21:55:14.990 INFO:teuthology.results:Sending results to jcollin@redhat.com: Test Run: jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/
failed:  8
dead:    11
running: 0
waiting: 0
queued:  0
passed:  81


Fail
=================================================================
[5833385]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:38:38
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833385/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833385/

    "2021-01-27T09:40:52.361223+0000 mds.c (mds.0) 39 : cluster [WRN] Scrub
    error on inode 0x10000007a73
    (/client.0/tmp/payload.1/multiple_rsync_payload.146341/firmware) see mds.c
    log and `damage ls` output for details" in cluster log

[5833395]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:15:17
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833395/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833395/

    "2021-01-27T09:45:17.570245+0000 mds.f (mds.0) 19 : cluster [WRN] Scrub
    error on inode 0x10000002ea8 (/client.0/tmp/tmp.mv9O2luNHA/p8) see mds.f
    log and `damage ls` output for details" in cluster log

[5833406]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:12:05
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833406/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833406/

    "2021-01-27T09:48:00.907089+0000 mds.f (mds.0) 14 : cluster [WRN] Scrub
    error on inode 0x100000002cd (/client.0/tmp/tmp) see mds.f log and `damage
    ls` output for details" in cluster log

[5833413]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:43:41
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833413/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833413/

    "2021-01-27T09:57:14.657968+0000 mds.c (mds.0) 14 : cluster [WRN] Scrub
    error on inode 0x100000001fe (/client.0/tmp/blogbench-1.0/src) see mds.c
    log and `damage ls` output for details" in cluster log

[5833414]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833414/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833414/
sentry: https://sentry.ceph.com/organizations/ceph/?query=382e00883dd24c81886e62fe8d6ff826

    Command failed on smithi031 with status 128: 'cd /lib/firmware/updates &&
    sudo git fetch origin && sudo git reset --hard origin/master'

[5833433]  fs/verify/{begin centos_latest clusters/1a5s-mds-1c-client conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   01:04:38
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833433/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833433/
sentry: https://sentry.ceph.com/organizations/ceph/?query=00e07d4bd6c84d019be177ca2b5fed08

    Command failed (workunit test suites/fsstress.sh) on smithi154 with status
    135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=164b0f632a5fe3384fa7cbb10929c65ed7cc0f12
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[5833435]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:31:35
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833435/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833435/

    "2021-01-27T10:18:30.830786+0000 mds.c (mds.0) 19 : cluster [WRN] Scrub
    error on inode 0x10000000263 (/client.0/tmp/testdir/dir1) see mds.c log and
    `damage ls` output for details" in cluster log

[5833446]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:14:13
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833446/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833446/

    "2021-01-27T10:34:41.847120+0000 mds.f (mds.1) 2 : cluster [WRN] Scrub
    error on inode 0x10000000572 (/client.0/tmp/fsstress/ltp-
    full-20091231/testcases/network/rpc) see mds.f log and `damage ls` output
    for details" in cluster log



Dead
=================================================================
[5833389]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   01:09:13
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833389/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833389/

    [Errno 113] No route to host

[5833403]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833403/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833403/

[5833404]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833404/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833404/

[5833410]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/fs/misc}}
-----------------------------------------------------------------
time:   01:09:41
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833410/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833410/
sentry: https://sentry.ceph.com/organizations/ceph/?query=de5e045ddd18457981b48a80a6984bfa

    SSH connection to smithi013 was lost: 'sudo rm -rf --
    /home/ubuntu/cephtest/workunits.list.client.0
    /home/ubuntu/cephtest/clone.client.0'

[5833416]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:51:00
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833416/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833416/

    SSH connection to smithi142 was lost: 'sudo rm /etc/logrotate.d/ceph-
    test.conf'

[5833432]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833432/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833432/

[5833438]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:39:57
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833438/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833438/

    SSH connection to smithi060 was lost: 'sudo rm /etc/logrotate.d/ceph-
    test.conf'

[5833439]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833439/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833439/

[5833440]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/asok_dump_tree}
-----------------------------------------------------------------
time:   00:51:59
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833440/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833440/

    SSH connection to smithi080 was lost: 'sudo rm /etc/logrotate.d/ceph-
    test.conf'

[5833453]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/iogen}}
-----------------------------------------------------------------
time:   01:09:07
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833453/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833453/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c4235ed58ece4a74837e7c82ac346a71

    SSH connection to smithi014 was lost: 'sudo rm -rf --
    /home/ubuntu/cephtest/workunits.list.client.0
    /home/ubuntu/cephtest/clone.client.0'

[5833461]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/fs/misc}}
-----------------------------------------------------------------
time:   01:08:57
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833461/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833461/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4e2291841e574bdabd6ada4d01ebc422

    Unable to open channel.



Pass
=================================================================
[5833367] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:12:37
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833367/

[5833368] fs/bugs/client_trim_caps/{begin clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/trim-i22073}
time:   00:06:50
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833368/

[5833369] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls}
time:   00:10:15
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833369/

[5833370] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client}
time:   00:07:28
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833370/

[5833371] fs/mirror/{begin cephfs-mirror/{one-per-cluster} clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{ubuntu_latest} tasks/mirror}
time:   00:48:09
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833371/

[5833372] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{frag_enable osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:28:19
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833372/

[5833373] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   00:19:06
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833373/

[5833374] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover}
time:   00:38:04
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833374/

[5833375] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:08:10
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833375/

[5833376] fs/shell/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cephfs-shell}
time:   00:32:13
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833376/

[5833377] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:38:42
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833377/

[5833378] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
time:   03:39:05
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833378/

[5833379] fs/top/{begin cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/whitelist_health supported-random-distros$/{centos_8} tasks/fstop}
time:   00:10:26
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833379/

[5833380] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:19:31
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833380/

[5833381] fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}}
time:   00:30:20
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833381/

[5833382] fs/verify/{begin centos_latest clusters/1a5s-mds-1c-client conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/dbench validater/lockdep}
time:   01:00:09
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833382/

[5833383] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}}
time:   00:46:53
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833383/

[5833384] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/direct_io}}
time:   00:12:35
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833384/

[5833386] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   02:14:17
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833386/

[5833387] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}}
time:   00:37:25
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833387/

[5833388] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:38:18
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833388/

[5833390] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/dbench}}
time:   00:57:20
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833390/

[5833391] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:19:59
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833391/

[5833392] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}}
time:   00:49:16
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833392/

[5833393] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:09:34
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833393/

[5833394] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/norstats}}
time:   00:15:44
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833394/

[5833396] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:10:42
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833396/

[5833397] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/fsx}}
time:   00:46:27
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833397/

[5833398] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:13:18
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833398/

[5833399] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}}
time:   00:12:18
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833399/

[5833400] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}}
time:   00:23:39
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833400/

[5833401] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   01:14:00
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833401/

[5833402] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin}
time:   00:25:56
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833402/

[5833405] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/iozone}}
time:   00:11:57
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833405/

[5833407] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:13:25
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833407/

[5833408] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/direct_io}}
time:   00:13:59
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833408/

[5833409] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:10:33
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833409/

[5833411] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}}
time:   02:00:48
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833411/

[5833412] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:18:23
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833412/

[5833415] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/dbench}}
time:   00:53:16
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833415/

[5833417] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:09:35
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833417/

[5833418] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/norstats}}
time:   00:14:59
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833418/

[5833419] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:50:53
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833419/

[5833420] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool}
time:   00:14:39
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833420/

[5833421] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}}
time:   00:15:43
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833421/

[5833422] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/fsx}}
time:   00:27:10
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833422/

[5833423] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:17:37
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833423/

[5833424] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}}
time:   00:10:54
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833424/

[5833425] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:09:18
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833425/

[5833426] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}}
time:   00:20:10
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833426/

[5833427] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/iogen}}
time:   00:58:52
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833427/

[5833428] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:12:05
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833428/

[5833429] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/iozone}}
time:   00:18:03
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833429/

[5833430] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:11:51
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833430/

[5833431] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/pjd}}
time:   00:12:49
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833431/

[5833434] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/direct_io}}
time:   00:13:16
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833434/

[5833436] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:05:39
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833436/

[5833437] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}}
time:   00:53:08
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833437/

[5833441] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/dbench}}
time:   01:10:45
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833441/

[5833442] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:14:54
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833442/

[5833443] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}}
time:   00:29:10
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833443/

[5833444] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:09:23
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833444/

[5833445] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/norstats}}
time:   00:16:33
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833445/

[5833447] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:19:06
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833447/

[5833448] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/fsx}}
time:   00:31:10
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833448/

[5833449] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:12:12
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833449/

[5833450] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}}
time:   00:10:43
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833450/

[5833451] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}}
time:   00:23:33
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833451/

[5833452] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   02:45:39
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833452/

[5833454] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:38:43
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833454/

[5833455] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/iozone}}
time:   00:12:51
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833455/

[5833456] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/pjd}}
time:   00:11:13
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833456/

[5833457] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:20:14
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833457/

[5833458] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair}
time:   00:09:21
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833458/

[5833459] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/direct_io}}
time:   00:14:50
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833459/

[5833460] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:09:44
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833460/

[5833462] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}}
time:   02:29:20
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833462/

[5833463] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:09:59
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833463/

[5833464] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}}
time:   00:21:28
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833464/

[5833465] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:12:54
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833465/

[5833466] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/dbench}}
time:   01:18:12
info:   http://pulpito.ceph.com/jcollin-2021-01-27_09:11:59-fs-wip-jcollin-dispatch-queue-throttle-test-distro-basic-smithi/5833466/