2020-06-14T11:48:19.427 INFO:root:teuthology version: 1.0.0-6c966fe 2020-06-14T11:48:19.429 INFO:teuthology.results:subset: '23/9999' 2020-06-14T11:48:19.429 INFO:teuthology.results:seed: '2169' 2020-06-14T12:04:38.685 INFO:root:teuthology version: 1.0.0-6c966fe 2020-06-14T12:04:38.705 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-06-14T16:31:44.114 INFO:teuthology.results:Tests finished! gathering results... 2020-06-14T16:31:44.203 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 20 Fail ================================================================= [5132407] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:57:23 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132407/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132407/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=0327e15f6cb5412bafbeb5c0cb8fd7ac Command failed (workunit test suites/fsstress.sh) on smithi067 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a784fb02df85e94236ad363d20082ae173687bc4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [5132408] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind} ----------------------------------------------------------------- time: 03:37:12 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132408/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132408/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=16d599fb83844b5289cda0f30a2a410c Command failed on smithi074 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp' [5132416] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind} ----------------------------------------------------------------- time: 04:05:59 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132416/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132416/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=1bd5e73ac5ce4b84a18b81bc9cbfe01d [5132424] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind} ----------------------------------------------------------------- time: 03:25:55 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132424/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132424/ sentry: http://sentry.ceph.com/sepia/teuthology/?q=029f04e9d0ac47efb5224a54c26c7a1f Pass ================================================================= [5132404] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd} time: 00:16:28 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132404/ [5132405] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep} time: 01:03:23 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132405/ [5132406] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports} time: 00:15:21 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132406/ [5132409] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots} time: 00:32:33 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132409/ [5132410] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build} time: 00:48:06 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132410/ [5132411] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd} time: 00:16:10 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132411/ [5132412] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep} time: 01:00:03 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132412/ [5132413] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc} time: 00:58:10 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132413/ [5132414] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats} time: 00:16:53 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132414/ [5132415] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd} time: 00:25:04 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132415/ [5132417] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench} time: 00:33:39 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132417/ [5132418] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench} time: 01:06:34 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132418/ [5132419] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress} time: 00:21:52 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132419/ [5132420] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep} time: 00:59:42 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132420/ [5132421] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb} time: 00:41:37 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132421/ [5132422] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress} time: 00:22:06 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132422/ [5132423] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd} time: 00:17:14 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132423/ [5132425] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx} time: 00:35:25 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132425/ [5132426] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd} time: 00:20:20 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132426/ [5132427] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress} time: 00:19:21 info: http://pulpito.ceph.com/teuthology-2020-06-10_04:15:02-multimds-master-testing-basic-smithi/5132427/