2020-10-28T04:16:27.302 INFO:root:teuthology version: 1.0.0-f4fe713
2020-10-28T04:16:27.304 INFO:teuthology.results:subset: '23/100000'
2020-10-28T04:16:27.305 INFO:teuthology.results:seed: '6274'
2020-10-28T04:36:57.419 INFO:root:teuthology version: 1.0.0-f4fe713
2020-10-28T04:36:57.453 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-10-28T09:21:02.874 INFO:teuthology.results:Tests finished! gathering results...
2020-10-28T09:21:05.610 INFO:scrape:Found 25 jobs
2020-10-28T09:21:08.279 INFO:scrape:Found 2 distinct failure reasons
2020-10-28T09:21:08.280 INFO:scrape:Failure: 
2020-10-28T09:21:08.280 INFO:scrape:3 jobs: ['5566933', '5566941', '5566949']
2020-10-28T09:21:08.281 INFO:scrape:suites intersection: ['centos_latest', 'clusters/3-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'multimds/verify/{begin', 'osd}', 'overrides/{fuse-default-perm-no', 'session_timeout', 'tasks/cfuse_workunit_suites_fsstress', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}']
2020-10-28T09:21:08.281 INFO:scrape:suites union: ['centos_latest', 'clusters/3-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'multimds/verify/{begin', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{fuse-default-perm-no', 'session_timeout', 'tasks/cfuse_workunit_suites_fsstress', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}']
2020-10-28T09:21:08.281 INFO:scrape:
2020-10-28T09:21:08.281 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi018 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=677d939b391f46fb54c3027ead30cfa1757cc778 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'
2020-10-28T09:21:08.282 INFO:scrape:1 jobs: ['5566948']
2020-10-28T09:21:08.282 INFO:scrape:suites: ['begin', 'ceph-thrash/mds', 'clusters/9-mds-3-standby', 'conf/{client', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multimds/thrash/{0-supported-random-distro$/{centos_8}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{fuse-default-perm-no', 'session_timeout', 'tasks/cfuse_workunit_suites_pjd}', 'thrash/{frag_enable', 'thrash_debug}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2020-10-28T09:21:08.282 INFO:scrape:
2020-10-28T09:21:08.284 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/
failed:  4
dead:    0
running: 0
waiting: 0
queued:  0
passed:  21


Fail
=================================================================
[5566933]  multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind}
-----------------------------------------------------------------
time:   04:31:59
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566933/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566933/
sentry: https://sentry.ceph.com/ceph/sepia/?query=9a7b29bfe9954e36932cb2169dd23cc1

[5566941]  multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind}
-----------------------------------------------------------------
time:   03:54:45
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566941/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566941/
sentry: https://sentry.ceph.com/ceph/sepia/?query=279e5ee15ec04fdaa74abb1057ee561e

[5566948]  multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
-----------------------------------------------------------------
time:   00:14:11
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566948/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566948/
sentry: https://sentry.ceph.com/ceph/sepia/?query=9f97e25fc74643e594fda36385877654

    Command failed (workunit test suites/pjd.sh) on smithi018 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=677d939b391f46fb54c3027ead30cfa1757cc778
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[5566949]  multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind}
-----------------------------------------------------------------
time:   03:38:45
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566949/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566949/
sentry: https://sentry.ceph.com/ceph/sepia/?query=9fa8459d3cfd4599bc516835eaca57f0



Pass
=================================================================
[5566926] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:48:19
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566926/

[5566927] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:15:21
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566927/

[5566928] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:31:26
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566928/

[5566929] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   01:40:05
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566929/

[5566930] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
time:   00:38:53
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566930/

[5566931] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:21:19
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566931/

[5566932] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_mds_metrics}
time:   00:16:26
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566932/

[5566934] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
time:   00:17:53
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566934/

[5566935] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:14:28
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566935/

[5566936] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   02:28:39
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566936/

[5566937] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   01:16:24
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566937/

[5566938] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   00:37:00
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566938/

[5566939] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:13:03
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566939/

[5566940] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:21:34
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566940/

[5566942] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
time:   00:49:01
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566942/

[5566943] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   01:16:14
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566943/

[5566944] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:17:59
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566944/

[5566945] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   00:56:43
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566945/

[5566946] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   01:53:24
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566946/

[5566947] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
time:   00:19:10
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566947/

[5566950] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:30:28
info:   http://pulpito.ceph.com/teuthology-2020-10-28_04:15:02-multimds-master-distro-basic-smithi/5566950/