2020-04-10T00:40:38.466 INFO:root:teuthology version: 1.0.0-0f93372 2020-04-10T00:40:38.466 INFO:teuthology.results:subset: '16/999' 2020-04-10T00:40:38.466 INFO:teuthology.results:seed: '4299' 2020-04-10T00:56:47.317 INFO:root:teuthology version: 1.0.0-0f93372 2020-04-10T00:56:47.329 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-04-10T06:27:53.143 INFO:teuthology.results:Tests finished! gathering results... 2020-04-10T06:27:53.355 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 21 Fail ================================================================= [4935387] multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml supported-random-distros$/{ubuntu_16.04.yaml} tasks/cfuse_workunit_misc.yaml} ----------------------------------------------------------------- time: 00:30:46 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935387/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935387/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=f54b411519544da3997dc5bbd5cf82a0 Command failed (workunit test fs/misc/rstats.sh) on smithi027 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e164b827b98f0db9596e9c457a841971af3d52be TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/rstats.sh' [4935390] multimds/verify/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} ----------------------------------------------------------------- time: 04:47:21 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935390/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935390/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=e10194e06aea40d99ee6f70b59566ea7 [4935383] multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml supported-random-distros$/{rhel_latest.yaml} tasks/cephfs_test_snapshots.yaml} ----------------------------------------------------------------- time: 00:28:12 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935383/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935383/ timed out Pass ================================================================= [4935372] multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml supported-random-distros$/{rhel_latest.yaml} tasks/cfuse_workunit_suites_ffsb.yaml} time: 00:36:45 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935372/ [4935373] multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} supported-random-distros$/{rhel_latest.yaml} tasks/cfuse_workunit_suites_pjd.yaml} time: 00:17:36 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935373/ [4935370] multimds/verify/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} time: 00:25:52 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935370/ [4935371] multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml supported-random-distros$/{ubuntu_latest.yaml} tasks/cfuse_workunit_suites_dbench.yaml} time: 01:30:13 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935371/ [4935376] multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml supported-random-distros$/{ubuntu_latest.yaml} tasks/cfuse_workunit_suites_fsx.yaml} time: 01:42:34 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935376/ [4935377] multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} supported-random-distros$/{centos_latest.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} time: 00:53:34 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935377/ [4935374] multimds/verify/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} time: 00:50:30 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935374/ [4935375] multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml supported-random-distros$/{ubuntu_16.04.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} time: 00:17:28 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935375/ [4935386] multimds/verify/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} time: 01:44:01 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935386/ [4935378] multimds/verify/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} time: 00:27:27 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935378/ [4935379] multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml supported-random-distros$/{ubuntu_16.04.yaml} tasks/cfuse_workunit_suites_pjd.yaml} time: 00:11:19 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935379/ [4935369] multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} supported-random-distros$/{ubuntu_16.04.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} time: 00:29:39 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935369/ [4935368] multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml supported-random-distros$/{centos_latest.yaml} tasks/cfuse_workunit_suites_blogbench.yaml} time: 00:50:37 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935368/ [4935381] multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} supported-random-distros$/{rhel_latest.yaml} tasks/cfuse_workunit_suites_pjd.yaml} time: 00:17:51 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935381/ [4935380] multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml supported-random-distros$/{ubuntu_16.04.yaml} tasks/cephfs_test_exports.yaml} time: 00:11:45 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935380/ [4935391] multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml supported-random-distros$/{rhel_latest.yaml} tasks/cfuse_workunit_suites_blogbench.yaml} time: 01:01:47 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935391/ [4935388] multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml supported-random-distros$/{ubuntu_16.04.yaml} tasks/cfuse_workunit_norstats.yaml} time: 00:18:18 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935388/ [4935384] multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml supported-random-distros$/{rhel_latest.yaml} tasks/cfuse_workunit_kernel_untar_build.yaml} time: 02:52:55 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935384/ [4935389] multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} supported-random-distros$/{rhel_latest.yaml} tasks/cfuse_workunit_suites_pjd.yaml} time: 00:18:41 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935389/ [4935385] multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} supported-random-distros$/{ubuntu_latest.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} time: 00:14:22 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935385/ [4935382] multimds/verify/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} time: 00:50:47 info: http://pulpito.ceph.com/teuthology-2020-04-08_14:15:04-multimds-mimic-testing-basic-smithi/4935382/ 2020-04-10T06:27:53.494 INFO:teuthology.results:starting coverage generation 2020-04-10T06:27:53.507 ERROR:teuthology.results:error generating memo/results Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 35, in main int(args['--timeout']), args['--dry-run']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 84, in results generate_coverage(archive_dir, name) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 107, in generate_coverage archive_dir, File "/usr/lib/python2.7/subprocess.py", line 711, in __init__ errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory