2020-05-04T09:47:37.221 INFO:root:teuthology version: 1.0.0-58684fc5
2020-05-04T09:47:37.223 INFO:teuthology.results:subset: '20/9999'
2020-05-04T09:47:37.223 INFO:teuthology.results:seed: '9753'
2020-05-04T10:09:45.108 INFO:root:teuthology version: 1.0.0-58684fc5
2020-05-04T10:09:45.128 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-05-04T21:53:58.879 INFO:teuthology.results:Tests finished! gathering results...
2020-05-04T21:53:58.942 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/
failed:  5
dead:    1
running: 0
waiting: 0
queued:  0
passed:  18


Fail
=================================================================
[5018750]  multimds/verify/{begin.yaml centos_latest.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{k-testing.yaml mount.yaml ms-die-on-skipped.yaml} objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml}
-----------------------------------------------------------------
time:   02:44:48
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018750/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018750/

    reached maximum tries (50) after waiting for 300 seconds

[5018760]  multimds/basic/{0-supported-random-distro$/{ubuntu_latest.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}
-----------------------------------------------------------------
time:   00:27:03
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018760/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018760/

    [Errno None] Unable to connect to port 22 on 172.21.15.57

[5018762]  multimds/thrash/{0-supported-random-distro$/{rhel_latest.yaml} begin.yaml ceph-thrash/mds.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}
-----------------------------------------------------------------
time:   00:26:21
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018762/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018762/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=9ffd04d00bba42b8a469da490c8fcf1c

    Command failed (workunit test suites/fsstress.sh) on smithi124 with status
    2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=7b78fa3712b750766e196b3d64d1ddbc858ba41c
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[5018766]  multimds/verify/{begin.yaml centos_latest.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{k-testing.yaml mount.yaml ms-die-on-skipped.yaml} objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml}
-----------------------------------------------------------------
time:   02:10:23
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018766/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018766/

    "2020-05-04T11:20:25.955632+0000 mds.b (mds.2) 1 : cluster [WRN]
    client.4391 isn't responding to mclientcaps(revoke), ino 0x30000002e3a
    pending pAsLsXs issued pAsLsXsFsx, sent 63.597036 seconds ago" in cluster
    log

[5018782]  multimds/verify/{begin.yaml centos_latest.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{k-testing.yaml mount.yaml ms-die-on-skipped.yaml} objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml}
-----------------------------------------------------------------
time:   02:23:00
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018782/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018782/

    "2020-05-04T11:45:45.389171+0000 mds.c (mds.1) 1 : cluster [WRN]
    client.4379 isn't responding to mclientcaps(revoke), ino 0x2000000334d
    pending pAsXsFs issued pAsXsFsx, sent 60.340307 seconds ago" in cluster log



Dead
=================================================================
[5018748]  multimds/basic/{0-supported-random-distro$/{centos_latest.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018748/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018748/



Pass
=================================================================
[5018740] multimds/basic/{0-supported-random-distro$/{rhel_latest.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}
time:   02:37:13
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018740/

[5018742] multimds/verify/{begin.yaml centos_latest.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml}
time:   00:55:15
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018742/

[5018744] multimds/basic/{0-supported-random-distro$/{centos_latest.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml}
time:   00:16:54
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018744/

[5018746] multimds/thrash/{0-supported-random-distro$/{centos_latest.yaml} begin.yaml ceph-thrash/mds.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}
time:   00:27:04
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018746/

[5018752] multimds/basic/{0-supported-random-distro$/{rhel_latest.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml}
time:   00:14:49
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018752/

[5018754] multimds/thrash/{0-supported-random-distro$/{rhel_latest.yaml} begin.yaml ceph-thrash/mon.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} msgr-failures/none.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml}
time:   00:15:43
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018754/

[5018756] multimds/basic/{0-supported-random-distro$/{centos_latest.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml}
time:   00:14:25
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018756/

[5018758] multimds/verify/{begin.yaml centos_latest.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml}
time:   00:54:04
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018758/

[5018764] multimds/basic/{0-supported-random-distro$/{ubuntu_latest.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml}
time:   02:40:24
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018764/

[5018768] multimds/basic/{0-supported-random-distro$/{ubuntu_latest.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml}
time:   00:45:27
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018768/

[5018770] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest.yaml} begin.yaml ceph-thrash/mon.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} msgr-failures/none.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml}
time:   00:13:18
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018770/

[5018772] multimds/basic/{0-supported-random-distro$/{centos_latest.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml}
time:   00:15:27
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018772/

[5018774] multimds/verify/{begin.yaml centos_latest.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml}
time:   00:59:18
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018774/

[5018776] multimds/basic/{0-supported-random-distro$/{rhel_latest.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml}
time:   00:36:15
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018776/

[5018778] multimds/thrash/{0-supported-random-distro$/{centos_latest.yaml} begin.yaml ceph-thrash/mds.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}
time:   00:27:07
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018778/

[5018780] multimds/basic/{0-supported-random-distro$/{centos_latest.yaml} begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml}
time:   00:57:32
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018780/

[5018784] multimds/basic/{0-supported-random-distro$/{centos_latest.yaml} begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}
time:   00:32:36
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018784/

[5018786] multimds/thrash/{0-supported-random-distro$/{rhel_latest.yaml} begin.yaml ceph-thrash/mon.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/stock/{k-stock.yaml rhel_8.yaml} ms-die-on-skipped.yaml}} msgr-failures/none.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml}
time:   00:16:05
info:   http://pulpito.ceph.com/teuthology-2020-05-03_13:15:03-multimds-octopus-testing-basic-smithi/5018786/
2020-05-04T21:53:59.025 INFO:teuthology.results:starting coverage generation
2020-05-04T21:53:59.031 ERROR:teuthology.results:error generating memo/results
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 35, in main
    int(args['--timeout']), args['--dry-run'])
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 84, in results
    generate_coverage(archive_dir, name)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 107, in generate_coverage
    archive_dir,
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-coverage': '/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-coverage'