2020-04-05T13:15:50.333 INFO:root:teuthology version: 1.0.0-37546b3
2020-04-05T13:15:50.334 INFO:teuthology.results:subset: '20/9999'
2020-04-05T13:15:50.334 INFO:teuthology.results:seed: '7575'
2020-04-05T13:15:50.457 INFO:root:teuthology version: 1.0.0-37546b3
2020-04-05T13:15:50.457 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-04-06T01:16:03.269 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2020-04-06T01:16:03.342 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/
failed:  3
dead:    0
running: 1
waiting: 0
queued:  0
passed:  20


Fail
=================================================================
[4926551]  multimds/basic/{0-supported-random-distro$/{centos_8.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml}
-----------------------------------------------------------------
time:   04:03:05
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926551/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926551/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=25f56e5ccc704cd89e0e9219ecff67dd

    Command failed (workunit test kernel_untar_build.sh) on smithi081 with
    status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=17cd5d7f924f61abc03850f6549554a4fa098003
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

[4926549]  multimds/basic/{0-supported-random-distro$/{centos_8.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}
-----------------------------------------------------------------
time:   00:26:51
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926549/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926549/

    timed out

[4926560]  multimds/verify/{begin.yaml centos_latest.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{k-testing.yaml mount.yaml ms-die-on-skipped.yaml} objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml}
-----------------------------------------------------------------
time:   02:33:21
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926560/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926560/

    "2020-04-05T14:51:10.773451+0000 mds.c (mds.1) 1 : cluster [WRN]
    client.4361 isn't responding to mclientcaps(revoke), ino 0x20000002efa
    pending pAsLsXs issued pAsLsXsFsx, sent 62.079811 seconds ago" in cluster
    log



Running
=================================================================
[4926543] multimds/basic/{0-supported-random-distro$/{centos_8.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml}
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926543/



Pass
=================================================================
[4926552] multimds/verify/{begin.yaml centos_latest.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{k-testing.yaml mount.yaml ms-die-on-skipped.yaml} objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml}
time:   02:15:34
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926552/

[4926554] multimds/thrash/{0-supported-random-distro$/{rhel_8.yaml} begin.yaml ceph-thrash/mon.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} msgr-failures/none.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml}
time:   00:14:20
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926554/

[4926553] multimds/basic/{0-supported-random-distro$/{ubuntu_latest.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml}
time:   00:43:10
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926553/

[4926562] multimds/thrash/{0-supported-random-distro$/{centos_8.yaml} begin.yaml ceph-thrash/mon.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} msgr-failures/none.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml}
time:   00:15:02
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926562/

[4926561] multimds/basic/{0-supported-random-distro$/{centos_8.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}
time:   00:33:10
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926561/

[4926555] multimds/basic/{0-supported-random-distro$/{rhel_8.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml}
time:   00:15:59
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926555/

[4926539] multimds/basic/{0-supported-random-distro$/{centos_8.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}
time:   01:27:51
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926539/

[4926545] multimds/basic/{0-supported-random-distro$/{ubuntu_latest.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml}
time:   00:15:36
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926545/

[4926544] multimds/verify/{begin.yaml centos_latest.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{k-testing.yaml mount.yaml ms-die-on-skipped.yaml} objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml}
time:   02:31:05
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926544/

[4926547] multimds/basic/{0-supported-random-distro$/{rhel_8.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml}
time:   00:14:24
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926547/

[4926546] multimds/thrash/{0-supported-random-distro$/{centos_8.yaml} begin.yaml ceph-thrash/mon.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} msgr-failures/none.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml}
time:   00:15:35
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926546/

[4926541] multimds/basic/{0-supported-random-distro$/{centos_8.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml}
time:   00:16:59
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926541/

[4926540] multimds/verify/{begin.yaml centos_latest.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml}
time:   00:53:28
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926540/

[4926550] multimds/thrash/{0-supported-random-distro$/{rhel_8.yaml} begin.yaml ceph-thrash/mds.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}
time:   00:20:46
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926550/

[4926542] multimds/thrash/{0-supported-random-distro$/{centos_8.yaml} begin.yaml ceph-thrash/mds.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}
time:   00:25:06
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926542/

[4926556] multimds/verify/{begin.yaml centos_latest.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml}
time:   00:57:59
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926556/

[4926558] multimds/thrash/{0-supported-random-distro$/{rhel_8.yaml} begin.yaml ceph-thrash/mds.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}
time:   00:28:32
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926558/

[4926559] multimds/basic/{0-supported-random-distro$/{rhel_8.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml}
time:   00:57:43
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926559/

[4926548] multimds/verify/{begin.yaml centos_latest.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml}
time:   00:52:44
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926548/

[4926557] multimds/basic/{0-supported-random-distro$/{ubuntu_latest.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml}
time:   00:31:30
info:   http://pulpito.ceph.com/teuthology-2020-04-05_13:15:03-multimds-octopus-testing-basic-smithi/4926557/
2020-04-06T01:16:03.452 INFO:teuthology.results:starting coverage generation
2020-04-06T01:16:03.461 ERROR:teuthology.results:error generating memo/results
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 35, in main
    int(args['--timeout']), args['--dry-run'])
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 84, in results
    generate_coverage(archive_dir, name)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 107, in generate_coverage
    archive_dir,
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory