2020-04-27T03:01:49.396 INFO:root:teuthology version: 1.0.0-6cf40593
2020-04-27T03:01:49.397 INFO:teuthology.results:subset: '13/9999'
2020-04-27T03:01:49.397 INFO:teuthology.results:seed: '2071'
2020-04-27T11:47:41.281 INFO:root:teuthology version: 1.0.0-6cf40593
2020-04-27T11:47:41.455 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-04-27T11:47:41.482 INFO:teuthology.results:Tests finished! gathering results...
2020-04-27T11:47:41.536 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/
failed:  7
dead:    0
running: 0
waiting: 0
queued:  0
passed:  17


Fail
=================================================================
[4986712]  multimds/verify/{begin.yaml centos_latest.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml}
-----------------------------------------------------------------
time:   01:04:12
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986712/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986712/

    reached maximum tries (50) after waiting for 300 seconds

[4986713]  multimds/basic/{0-supported-random-distro$/{centos_latest.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}
-----------------------------------------------------------------
time:   00:26:39
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986713/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986713/

    timed out

[4986715]  multimds/thrash/{0-supported-random-distro$/{centos_latest.yaml} begin.yaml ceph-thrash/mds.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml}
-----------------------------------------------------------------
time:   00:16:46
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986715/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986715/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=652df77af61a42649b827c1b5c21cf89

    Command failed (workunit test suites/pjd.sh) on smithi081 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=a8c125c7d78f5cd973863993d258cd717ade4c99
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[4986727]  multimds/verify/{begin.yaml centos_latest.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{k-testing.yaml mount.yaml ms-die-on-skipped.yaml} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml}
-----------------------------------------------------------------
time:   03:20:48
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986727/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986727/

    "2020-04-27T04:06:26.856403+0000 mon.a (mon.0) 377 : cluster [WRN] Health
    check failed: Degraded data redundancy: 19/17086 objects degraded (0.111%),
    1 pg degraded (PG_DEGRADED)" in cluster log

[4986743]  multimds/verify/{begin.yaml centos_latest.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{k-testing.yaml mount.yaml ms-die-on-skipped.yaml} objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml}
-----------------------------------------------------------------
time:   03:05:54
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986743/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986743/

    saw valgrind issues

[4986745]  multimds/basic/{0-supported-random-distro$/{centos_latest.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml}
-----------------------------------------------------------------
time:   00:24:21
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986745/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986745/

    Command failed on smithi129 with status 1: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster
    ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok perf dump'

[4986757]  multimds/basic/{0-supported-random-distro$/{rhel_latest.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}
-----------------------------------------------------------------
time:   00:25:34
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986757/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986757/

    timed out



Pass
=================================================================
[4986717] multimds/basic/{0-supported-random-distro$/{ubuntu_latest.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml}
time:   02:47:07
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986717/

[4986719] multimds/verify/{begin.yaml centos_latest.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml}
time:   00:54:24
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986719/

[4986721] multimds/basic/{0-supported-random-distro$/{centos_latest.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml}
time:   00:41:41
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986721/

[4986723] multimds/thrash/{0-supported-random-distro$/{centos_latest.yaml} begin.yaml ceph-thrash/mon.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} msgr-failures/none.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}
time:   00:14:58
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986723/

[4986725] multimds/basic/{0-supported-random-distro$/{centos_latest.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml}
time:   00:15:26
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986725/

[4986729] multimds/basic/{0-supported-random-distro$/{ubuntu_latest.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml}
time:   00:35:27
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986729/

[4986731] multimds/thrash/{0-supported-random-distro$/{rhel_latest.yaml} begin.yaml ceph-thrash/mds.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}
time:   00:22:52
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986731/

[4986733] multimds/basic/{0-supported-random-distro$/{ubuntu_latest.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml}
time:   00:53:03
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986733/

[4986735] multimds/verify/{begin.yaml centos_latest.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml}
time:   00:58:26
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986735/

[4986737] multimds/basic/{0-supported-random-distro$/{rhel_latest.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}
time:   01:58:44
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986737/

[4986739] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest.yaml} begin.yaml ceph-thrash/mon.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml}
time:   00:15:20
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986739/

[4986741] multimds/basic/{0-supported-random-distro$/{centos_latest.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml}
time:   00:15:48
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986741/

[4986747] multimds/thrash/{0-supported-random-distro$/{rhel_latest.yaml} begin.yaml ceph-thrash/mds.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} msgr-failures/none.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}
time:   00:14:31
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986747/

[4986749] multimds/basic/{0-supported-random-distro$/{rhel_latest.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml}
time:   00:17:51
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986749/

[4986751] multimds/verify/{begin.yaml centos_latest.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml}
time:   00:55:03
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986751/

[4986753] multimds/basic/{0-supported-random-distro$/{centos_latest.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml}
time:   00:16:09
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986753/

[4986755] multimds/thrash/{0-supported-random-distro$/{rhel_latest.yaml} begin.yaml ceph-thrash/mon.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml}
time:   00:14:24
info:   http://pulpito.ceph.com/teuthology-2020-04-26_13:15:04-multimds-octopus-testing-basic-smithi/4986755/
2020-04-27T11:47:41.655 INFO:teuthology.results:starting coverage generation
2020-04-27T11:47:41.659 ERROR:teuthology.results:error generating memo/results
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 35, in main
    int(args['--timeout']), args['--dry-run'])
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 84, in results
    generate_coverage(archive_dir, name)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 107, in generate_coverage
    archive_dir,
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-coverage': '/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-coverage'