2020-10-04T17:30:55.684 INFO:root:teuthology version: 1.0.0-75888fc
2020-10-04T17:30:55.685 INFO:teuthology.results:subset: '22/9999'
2020-10-04T17:30:55.685 INFO:teuthology.results:seed: '3851'
2020-10-04T17:51:15.471 INFO:root:teuthology version: 1.0.0-75888fc
2020-10-04T17:51:15.507 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-10-04T21:41:19.490 INFO:teuthology.results:Tests finished! gathering results...
2020-10-04T21:41:22.292 INFO:scrape:Found 24 jobs
2020-10-04T21:41:22.569 INFO:scrape:Found 2 distinct failure reasons
2020-10-04T21:41:22.570 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/blogbench.sh
2020-10-04T21:41:22.570 INFO:scrape:1 jobs: ['5477853']
2020-10-04T21:41:22.570 INFO:scrape:suites: ['begin', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{rhel_latest}', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/stock/{k-stock', 'q_check_counter/check_counter', 'rhel_8}', 'tasks/cfuse_workunit_suites_blogbench}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2020-10-04T21:41:22.571 INFO:scrape:
2020-10-04T21:41:22.571 INFO:scrape:Failure: "2020-10-04T18:18:13.029822+0000 mds.l (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi047: (4627), after waiting 46.111 seconds during MDS startup" in cluster log
2020-10-04T21:41:22.571 INFO:scrape:1 jobs: ['5477850']
2020-10-04T21:41:22.572 INFO:scrape:suites: ['begin', 'ceph-thrash/mds', 'clusters/9-mds-3-standby', 'conf/{client', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multimds/thrash/{0-supported-random-distro$/{centos_latest}', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{fuse-default-perm-no', 'tasks/cfuse_workunit_suites_fsstress}', 'thrash/{frag_enable', 'thrash_debug}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2020-10-04T21:41:22.572 INFO:scrape:
2020-10-04T21:41:22.573 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/
failed:  2
dead:    0
running: 0
waiting: 0
queued:  0
passed:  22


Fail
=================================================================
[5477850]  multimds/thrash/{0-supported-random-distro$/{centos_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   02:27:55
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477850/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477850/

    "2020-10-04T18:18:13.029822+0000 mds.l (mds.0) 1 : cluster [WRN] evicting
    unresponsive client smithi047: (4627), after waiting 46.111 seconds during
    MDS startup" in cluster log

[5477853]  multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
-----------------------------------------------------------------
time:   03:25:02
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477853/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477853/
sentry: https://sentry.ceph.com/sentry/sepia/?query=77cac51250c0456c9dd74bc0a239d15e

    Command failed (workunit test suites/blogbench.sh) on smithi033 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ba4b96571dbe6582b442dc5c550a2b400bf4edc2
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh'



Pass
=================================================================
[5477834] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:23:54
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477834/

[5477836] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
time:   00:13:44
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477836/

[5477838] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:12:12
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477838/

[5477840] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:22:14
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477840/

[5477842] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
time:   00:39:14
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477842/

[5477844] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
time:   00:57:11
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477844/

[5477845] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
time:   00:24:50
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477845/

[5477846] multimds/thrash/{0-supported-random-distro$/{centos_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:14:37
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477846/

[5477847] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   01:08:53
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477847/

[5477848] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
time:   00:18:09
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477848/

[5477849] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   00:40:27
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477849/

[5477851] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:12:58
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477851/

[5477852] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
time:   00:55:59
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477852/

[5477854] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:12:59
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477854/

[5477855] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   00:58:30
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477855/

[5477856] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
time:   00:16:46
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477856/

[5477857] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   01:36:54
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477857/

[5477858] multimds/thrash/{0-supported-random-distro$/{rhel_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:22:52
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477858/

[5477859] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
time:   00:16:46
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477859/

[5477860] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
time:   00:49:28
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477860/

[5477861] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:34:49
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477861/

[5477862] multimds/thrash/{0-supported-random-distro$/{rhel_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:13:35
info:   http://pulpito.ceph.com/teuthology-2020-09-28_13:15:03-multimds-octopus-distro-basic-smithi/5477862/