2020-05-01T01:15:23.055 INFO:root:teuthology version: 1.0.0-58684fc5
2020-05-01T01:15:23.057 INFO:teuthology.results:subset: '8/2999'
2020-05-01T01:15:23.058 INFO:teuthology.results:seed: '8573'
2020-05-01T01:41:32.917 INFO:root:teuthology version: 1.0.0-58684fc5
2020-05-01T01:41:32.928 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-05-01T04:55:36.174 INFO:teuthology.results:Tests finished! gathering results...
2020-05-01T04:55:36.227 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/
failed:  4
dead:    0
running: 0
waiting: 0
queued:  0
passed:  20


Fail
=================================================================
[4991403]  multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/rhel/{k-distro.yaml rhel_latest.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}
-----------------------------------------------------------------
time:   00:40:16
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991403/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991403/

    "2020-05-01 02:07:20.708077 mon.a (mon.0) 224 : cluster [WRN] Health check
    failed: 2 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)" in cluster
    log

[4991407]  multimds/verify/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{kernel-testing.yaml mount.yaml ms-die-on-skipped.yaml} objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml}
-----------------------------------------------------------------
time:   02:56:36
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991407/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991407/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=5aa3d5b67e3549caaf4d1397540afd86

[4991420]  multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/random/{k-testing.yaml supported$/{ubuntu_16.04.yaml}} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}
-----------------------------------------------------------------
time:   00:23:50
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991420/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991420/

    [Errno None] Unable to connect to port 22 on 172.21.15.57

[4991422]  multimds/verify/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{kernel-testing.yaml mount.yaml ms-die-on-skipped.yaml} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml}
-----------------------------------------------------------------
time:   00:58:56
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991422/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991422/

    reached maximum tries (50) after waiting for 300 seconds



Pass
=================================================================
[4991405] multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml}
time:   00:23:36
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991405/

[4991409] multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}
time:   00:28:24
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991409/

[4991411] multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/random/{k-testing.yaml supported$/{rhel_latest.yaml}} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml}
time:   00:27:39
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991411/

[4991413] multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/rhel/{k-distro.yaml rhel_latest.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml}
time:   00:18:27
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991413/

[4991415] multimds/verify/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml}
time:   00:21:09
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991415/

[4991416] multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/random/{k-testing.yaml supported$/{rhel_latest.yaml}} ms-die-on-skipped.yaml}} msgr-failures/none.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml}
time:   00:17:55
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991416/

[4991418] multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml}
time:   00:14:33
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991418/

[4991425] multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/rhel/{k-distro.yaml rhel_latest.yaml} ms-die-on-skipped.yaml}} msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}
time:   00:25:46
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991425/

[4991427] multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/rhel/{k-distro.yaml rhel_latest.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml}
time:   00:58:53
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991427/

[4991429] multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml}
time:   01:06:11
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991429/

[4991431] multimds/verify/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml}
time:   00:21:02
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991431/

[4991433] multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml}
time:   00:14:37
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991433/

[4991434] multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient/{mount.yaml overrides/{distro/random/{k-testing.yaml supported$/{ubuntu_latest.yaml}} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml}
time:   00:16:37
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991434/

[4991437] multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/rhel/{k-distro.yaml rhel_latest.yaml} ms-die-on-skipped.yaml}} objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml}
time:   00:31:26
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991437/

[4991438] multimds/verify/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{kernel-testing.yaml mount.yaml ms-die-on-skipped.yaml} objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml}
time:   00:54:07
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991438/

[4991440] multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/random/{k-testing.yaml supported$/{rhel_latest.yaml}} ms-die-on-skipped.yaml}} msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}
time:   00:26:32
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991440/

[4991442] multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml}
time:   00:52:34
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991442/

[4991444] multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/random/{k-testing.yaml supported$/{rhel_latest.yaml}} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}
time:   01:47:19
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991444/

[4991447] multimds/verify/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml}
time:   00:18:36
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991447/

[4991448] multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient/{mount.yaml overrides/{distro/rhel/{k-distro.yaml rhel_latest.yaml} ms-die-on-skipped.yaml}} msgr-failures/none.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml}
time:   00:18:16
info:   http://pulpito.ceph.com/teuthology-2020-04-27_13:15:03-multimds-nautilus-testing-basic-smithi/4991448/
2020-05-01T04:55:36.292 INFO:teuthology.results:starting coverage generation
2020-05-01T04:55:36.295 ERROR:teuthology.results:error generating memo/results
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 35, in main
    int(args['--timeout']), args['--dry-run'])
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 84, in results
    generate_coverage(archive_dir, name)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 107, in generate_coverage
    archive_dir,
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-coverage': '/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-coverage'