2020-06-19T08:27:06.362 INFO:root:teuthology version: 1.0.0-4ff7700
2020-06-19T08:27:06.363 INFO:teuthology.results:subset: '3/8'
2020-06-19T08:27:06.364 INFO:teuthology.results:seed: '8632'
2020-06-19T11:11:52.696 INFO:root:teuthology version: 1.0.0-4ff7700
2020-06-19T11:11:52.722 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-06-19T20:58:08.383 INFO:teuthology.results:Tests finished! gathering results...
2020-06-19T20:58:08.626 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/
failed:  19
dead:    1
running: 0
waiting: 0
queued:  0
passed:  140


Fail
=================================================================
[5163056]  multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
-----------------------------------------------------------------
time:   00:13:56
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163056/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163056/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=6cf1ac49666149dda866efedfe1b5cb0

    Test failure: test_multimds_hardlink
    (tasks.cephfs.test_snapshots.TestSnapshots)

[5163071]  multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
-----------------------------------------------------------------
time:   01:01:45
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163071/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163071/

    reached maximum tries (50) after waiting for 300 seconds

[5163072]  multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
-----------------------------------------------------------------
time:   00:14:49
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163072/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163072/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=6394c5bf799d437aa652ce6dfeaa00fe

    Test failure: test_multimds_hardlink
    (tasks.cephfs.test_snapshots.TestSnapshots)

[5163078]  multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:30:14
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163078/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163078/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=42487951f3f7482dbcce2bb056f62a27

    Command failed (workunit test suites/fsstress.sh) on smithi193 with status
    2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=2ce9f32f2db8ec7e3ef96b2ab2ed9c18d986ebfa
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[5163080]  multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
-----------------------------------------------------------------
time:   01:20:09
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163080/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163080/

    "2020-06-19T09:30:19.611630+0000 mds.c (mds.2) 1 : cluster [WRN] evicting
    unresponsive client  (4716), after 302.066 seconds" in cluster log

[5163088]  multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
-----------------------------------------------------------------
time:   00:17:54
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163088/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163088/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=384bc956606f48318ad14c0eaa65dd5f

    Test failure: test_multimds_hardlink
    (tasks.cephfs.test_snapshots.TestSnapshots)

[5163093]  multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
-----------------------------------------------------------------
time:   03:22:53
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163093/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163093/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=ef0cbe8c350b4f2387ad01dcb3fde40a

    Command failed (workunit test suites/blogbench.sh) on smithi058 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=2ce9f32f2db8ec7e3ef96b2ab2ed9c18d986ebfa
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/blogbench.sh'

[5163104]  multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
-----------------------------------------------------------------
time:   00:25:52
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163104/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163104/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=0d4450214c894d438d45e2dc306e4121

    Test failure: test_multimds_hardlink
    (tasks.cephfs.test_snapshots.TestSnapshots)

[5163110]  multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   02:54:33
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163110/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163110/

    "2020-06-19T09:29:47.009330+0000 mds.h (mds.0) 1 : cluster [WRN] evicting
    unresponsive client smithi077: (4478), after waiting 49.425 seconds during
    MDS startup" in cluster log

[5163120]  multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
-----------------------------------------------------------------
time:   00:13:39
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163120/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163120/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=cac4ed0f7f764634bab6e83bbc2d66ec

    Test failure: test_multimds_hardlink
    (tasks.cephfs.test_snapshots.TestSnapshots)

[5163126]  multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:18:56
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163126/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163126/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=31293e934db44d778dde9c92a4034011

    Command failed (workunit test suites/fsstress.sh) on smithi085 with status
    2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=2ce9f32f2db8ec7e3ef96b2ab2ed9c18d986ebfa
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[5163135]  multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:06:25
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163135/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163135/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=50335dcb173740ccbefa7cda69ac29db

    Command failed on smithi172 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2
    librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-
    mirror rbd-nbd ceph-base-debuginfo ceph-common-debuginfo ceph-immutable-
    object-cache-debuginfo ceph-radosgw-debuginfo ceph-test-debuginfo ceph-
    base-debuginfo ceph-mgr-debuginfo ceph-mds-debuginfo ceph-mon-debuginfo
    ceph-osd-debuginfo ceph-fuse-debuginfo librados-devel-debuginfo
    libcephfs2-debuginfo librados2-debuginfo librbd1-debuginfo python3-cephfs-
    debuginfo python3-rados-debuginfo python3-rbd-debuginfo python3-rgw-
    debuginfo rbd-fuse-debuginfo rbd-mirror-debuginfo rbd-nbd-debuginfo
    python3-cephfs bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel
    NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel
    libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool
    libuuid-devel xfsprogs-devel'

[5163136]  multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
-----------------------------------------------------------------
time:   00:14:34
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163136/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163136/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=e1fb02c75b294fa09f35d72d987fcffb

    Test failure: test_multimds_hardlink
    (tasks.cephfs.test_snapshots.TestSnapshots)

[5163149]  multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
-----------------------------------------------------------------
time:   00:15:43
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163149/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163149/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=f5711005fd554565b7d11e810c077cf1

    Test failure: test_ephemeral_pin_dist_override_after
    (tasks.cephfs.test_exports.TestEphemeralPins)

[5163152]  multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
-----------------------------------------------------------------
time:   00:17:59
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163152/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163152/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=5ee3c6bdc38e43798d3034bdd6af53b6

    Test failure: test_multimds_hardlink
    (tasks.cephfs.test_snapshots.TestSnapshots)

[5163168]  multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
-----------------------------------------------------------------
time:   00:20:58
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163168/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163168/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=17e0fa9c24ed4930a0de9191d05b6716

    Test failure: test_multimds_hardlink
    (tasks.cephfs.test_snapshots.TestSnapshots)

[5163184]  multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
-----------------------------------------------------------------
time:   00:13:26
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163184/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163184/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=d3bdc9270ee54e3993428e85271361f7

    Test failure: test_multimds_hardlink
    (tasks.cephfs.test_snapshots.TestSnapshots)

[5163199]  multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
-----------------------------------------------------------------
time:   01:01:42
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163199/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163199/

    "2020-06-19T11:31:57.093359+0000 mon.a (mon.0) 195 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg inactive, 1 pg peering
    (PG_AVAILABILITY)" in cluster log

[5163200]  multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
-----------------------------------------------------------------
time:   00:15:33
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163200/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163200/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=51d8d4cabfca4c32849d0a1d9893bc7f

    Test failure: test_multimds_hardlink
    (tasks.cephfs.test_snapshots.TestSnapshots)



Dead
=================================================================
[5163094]  multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163094/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163094/



Pass
=================================================================
[5163053] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
time:   00:42:03
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163053/

[5163054] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:24:29
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163054/

[5163055] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
time:   00:18:20
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163055/

[5163057] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   02:27:42
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163057/

[5163058] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:17:31
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163058/

[5163059] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   01:22:25
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163059/

[5163060] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:15:32
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163060/

[5163061] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
time:   00:33:17
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163061/

[5163062] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:23:17
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163062/

[5163063] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   01:09:37
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163063/

[5163064] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   00:36:24
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163064/

[5163065] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
time:   00:20:10
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163065/

[5163066] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:14:29
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163066/

[5163067] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:23:58
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163067/

[5163068] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:12:33
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163068/

[5163069] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
time:   00:45:21
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163069/

[5163070] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:23:17
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163070/

[5163073] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   01:11:47
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163073/

[5163074] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:16:41
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163074/

[5163075] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   00:46:22
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163075/

[5163076] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:18:15
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163076/

[5163077] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
time:   00:41:06
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163077/

[5163079] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   00:54:56
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163079/

[5163081] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
time:   00:20:44
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163081/

[5163082] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:14:51
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163082/

[5163083] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:26:29
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163083/

[5163084] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:14:54
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163084/

[5163085] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
time:   00:45:25
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163085/

[5163086] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:28:22
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163086/

[5163087] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
time:   00:19:17
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163087/

[5163089] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   00:55:25
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163089/

[5163090] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:17:10
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163090/

[5163091] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   00:50:13
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163091/

[5163092] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:20:40
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163092/

[5163095] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   01:02:35
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163095/

[5163096] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   00:30:50
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163096/

[5163097] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
time:   00:16:29
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163097/

[5163098] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:14:30
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163098/

[5163099] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:42:47
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163099/

[5163100] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:15:43
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163100/

[5163101] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
time:   00:42:04
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163101/

[5163102] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:21:21
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163102/

[5163103] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
time:   00:57:48
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163103/

[5163105] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   00:50:10
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163105/

[5163106] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:17:26
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163106/

[5163107] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   01:00:27
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163107/

[5163108] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:15:35
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163108/

[5163109] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
time:   00:29:14
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163109/

[5163111] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   01:01:28
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163111/

[5163112] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   00:30:15
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163112/

[5163113] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
time:   00:20:18
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163113/

[5163114] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:13:31
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163114/

[5163115] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:29:37
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163115/

[5163116] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:17:45
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163116/

[5163117] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
time:   00:42:07
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163117/

[5163118] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:27:02
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163118/

[5163119] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
time:   00:18:39
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163119/

[5163121] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   02:27:57
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163121/

[5163122] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:17:01
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163122/

[5163123] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   00:47:19
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163123/

[5163124] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:18:15
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163124/

[5163125] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
time:   00:32:01
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163125/

[5163127] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   00:59:18
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163127/

[5163128] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   02:09:36
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163128/

[5163129] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
time:   00:18:57
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163129/

[5163130] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:12:38
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163130/

[5163131] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:36:23
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163131/

[5163132] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:12:27
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163132/

[5163133] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
time:   00:45:00
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163133/

[5163134] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:18:05
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163134/

[5163137] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   00:44:45
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163137/

[5163138] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:20:31
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163138/

[5163139] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   01:08:01
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163139/

[5163140] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:17:10
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163140/

[5163141] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
time:   00:30:56
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163141/

[5163142] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:16:17
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163142/

[5163143] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   00:55:43
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163143/

[5163144] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   00:36:11
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163144/

[5163145] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
time:   00:24:06
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163145/

[5163146] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:16:42
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163146/

[5163147] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:21:51
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163147/

[5163148] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:13:55
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163148/

[5163150] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:13:23
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163150/

[5163151] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
time:   00:20:15
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163151/

[5163153] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   01:31:46
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163153/

[5163154] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:17:56
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163154/

[5163155] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   00:46:13
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163155/

[5163156] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:15:33
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163156/

[5163157] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
time:   00:45:05
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163157/

[5163158] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:15:45
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163158/

[5163159] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   01:03:46
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163159/

[5163160] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   01:43:43
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163160/

[5163161] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
time:   00:20:27
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163161/

[5163162] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:15:42
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163162/

[5163163] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:26:50
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163163/

[5163164] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:14:20
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163164/

[5163165] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
time:   00:42:02
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163165/

[5163166] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:12:29
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163166/

[5163167] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
time:   00:56:24
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163167/

[5163169] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   00:43:46
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163169/

[5163170] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:19:11
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163170/

[5163171] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   00:48:04
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163171/

[5163172] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:16:33
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163172/

[5163173] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
time:   00:23:18
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163173/

[5163174] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:15:08
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163174/

[5163175] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   01:08:53
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163175/

[5163176] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   00:31:39
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163176/

[5163177] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
time:   00:16:38
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163177/

[5163178] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:17:14
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163178/

[5163179] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:47:09
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163179/

[5163180] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:17:32
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163180/

[5163181] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
time:   00:42:55
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163181/

[5163182] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:18:02
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163182/

[5163183] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
time:   00:18:40
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163183/

[5163185] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   02:18:30
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163185/

[5163186] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:20:08
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163186/

[5163187] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   01:16:04
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163187/

[5163188] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:17:13
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163188/

[5163189] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
time:   00:37:45
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163189/

[5163190] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:16:04
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163190/

[5163191] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   01:09:48
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163191/

[5163192] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   00:37:01
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163192/

[5163193] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
time:   00:22:19
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163193/

[5163194] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:14:53
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163194/

[5163195] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:28:36
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163195/

[5163196] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:14:33
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163196/

[5163197] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
time:   00:45:40
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163197/

[5163198] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:17:19
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163198/

[5163201] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   00:57:49
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163201/

[5163202] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:18:44
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163202/

[5163203] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   00:50:29
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163203/

[5163204] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:19:51
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163204/

[5163205] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
time:   00:31:40
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163205/

[5163206] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:16:04
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163206/

[5163207] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   00:56:16
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163207/

[5163208] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   01:13:17
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163208/

[5163209] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
time:   00:20:22
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163209/

[5163210] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:15:17
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163210/

[5163211] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:34:11
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163211/

[5163212] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:13:22
info:   http://pulpito.ceph.com/pdonnell-2020-06-19_04:47:54-multimds-wip-pdonnell-testing-20200619.024956-distro-basic-smithi/5163212/