2020-04-10T00:26:08.999 INFO:root:teuthology version: 1.0.0-0f93372
2020-04-10T00:26:08.999 INFO:teuthology.results:subset: '17/9999'
2020-04-10T00:26:08.999 INFO:teuthology.results:seed: '6271'
2020-04-10T00:40:38.057 INFO:root:teuthology version: 1.0.0-0f93372
2020-04-10T00:40:38.073 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-04-10T08:32:46.475 INFO:teuthology.results:Tests finished! gathering results...
2020-04-10T08:32:46.549 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/
failed:  6
dead:    0
running: 0
waiting: 0
queued:  0
passed:  18


Fail
=================================================================
[4935337]  multimds/basic/{0-supported-random-distro$/{ubuntu_latest.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml}
-----------------------------------------------------------------
time:   00:19:53
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935337/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935337/

    Command failed on smithi019 with status 1: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster
    ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok perf dump'

[4935351]  multimds/verify/{begin.yaml centos_latest.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml}
-----------------------------------------------------------------
time:   03:32:27
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935351/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935351/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=e87015b9504245e78bc9057e7a10c99e

[4935343]  multimds/verify/{begin.yaml centos_latest.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml}
-----------------------------------------------------------------
time:   03:44:26
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935343/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935343/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=d82b5f54a7744fa5869df76887013a12

[4935328]  multimds/basic/{0-supported-random-distro$/{rhel_8.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml}
-----------------------------------------------------------------
time:   03:14:11
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935328/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935328/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=bf576bf02e3b4283be1e4501f4e6333f

    Command failed (workunit test fs/norstats/kernel_untar_tar.sh) on smithi040
    with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=2e8037db723b4092f04ab4b49505d494dc66e949
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/fs/norstats/kernel_untar_tar.sh'

[4935335]  multimds/verify/{begin.yaml centos_latest.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml}
-----------------------------------------------------------------
time:   03:25:20
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935335/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935335/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=75a3b0b6a8144cc98ffdbfc558a76e4d

[4935344]  multimds/basic/{0-supported-random-distro$/{ubuntu_latest.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}
-----------------------------------------------------------------
time:   00:27:13
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935344/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935344/

    timed out



Pass
=================================================================
[4935350] multimds/thrash/{0-supported-random-distro$/{centos_8.yaml} begin.yaml ceph-thrash/mon.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}
time:   00:21:54
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935350/

[4935347] multimds/verify/{begin.yaml centos_latest.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{k-testing.yaml mount.yaml ms-die-on-skipped.yaml} objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml}
time:   01:20:46
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935347/

[4935341] multimds/basic/{0-supported-random-distro$/{ubuntu_latest.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml}
time:   00:18:44
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935341/

[4935346] multimds/thrash/{0-supported-random-distro$/{rhel_8.yaml} begin.yaml ceph-thrash/mds.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} msgr-failures/none.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml}
time:   00:14:09
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935346/

[4935336] multimds/basic/{0-supported-random-distro$/{rhel_8.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml}
time:   00:19:09
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935336/

[4935349] multimds/basic/{0-supported-random-distro$/{centos_8.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml}
time:   00:20:28
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935349/

[4935348] multimds/basic/{0-supported-random-distro$/{centos_8.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml}
time:   01:27:51
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935348/

[4935345] multimds/basic/{0-supported-random-distro$/{centos_8.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml}
time:   00:51:33
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935345/

[4935338] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest.yaml} begin.yaml ceph-thrash/mds.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml}
time:   00:14:23
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935338/

[4935339] multimds/verify/{begin.yaml centos_latest.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{k-testing.yaml mount.yaml ms-die-on-skipped.yaml} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml}
time:   01:10:09
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935339/

[4935329] multimds/basic/{0-supported-random-distro$/{ubuntu_latest.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml}
time:   00:38:51
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935329/

[4935342] multimds/thrash/{0-supported-random-distro$/{rhel_8.yaml} begin.yaml ceph-thrash/mon.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}
time:   00:23:49
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935342/

[4935334] multimds/thrash/{0-supported-random-distro$/{centos_8.yaml} begin.yaml ceph-thrash/mon.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}
time:   00:22:29
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935334/

[4935340] multimds/basic/{0-supported-random-distro$/{ubuntu_latest.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml}
time:   00:14:45
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935340/

[4935332] multimds/basic/{0-supported-random-distro$/{rhel_8.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml}
time:   00:59:43
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935332/

[4935333] multimds/basic/{0-supported-random-distro$/{centos_8.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}
time:   00:46:24
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935333/

[4935330] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest.yaml} begin.yaml ceph-thrash/mds.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} msgr-failures/none.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml}
time:   00:13:49
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935330/

[4935331] multimds/verify/{begin.yaml centos_latest.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{k-testing.yaml mount.yaml ms-die-on-skipped.yaml} objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml}
time:   00:55:59
info:   http://pulpito.ceph.com/teuthology-2020-04-08_13:15:04-multimds-octopus-testing-basic-smithi/4935331/
2020-04-10T08:32:46.646 INFO:teuthology.results:starting coverage generation
2020-04-10T08:32:46.653 ERROR:teuthology.results:error generating memo/results
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 35, in main
    int(args['--timeout']), args['--dry-run'])
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 84, in results
    generate_coverage(archive_dir, name)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 107, in generate_coverage
    archive_dir,
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory