2021-02-25T08:04:37.541 INFO:root:teuthology version: 1.1.0-9a6b849c
2021-02-25T08:04:37.544 INFO:teuthology.results:subset: '111/120000'
2021-02-25T08:04:37.545 INFO:teuthology.results:seed: '9562'
2021-02-25T08:38:43.559 INFO:root:teuthology version: 1.1.0-9a6b849c
2021-02-25T08:38:44.911 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-02-25T12:30:49.371 INFO:teuthology.results:Tests finished! gathering results...
2021-02-25T12:30:52.887 INFO:scrape:Found 24 jobs
2021-02-25T12:30:55.993 INFO:scrape:Found 4 distinct failure reasons
2021-02-25T12:30:55.994 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds
2021-02-25T12:30:55.994 INFO:scrape:1 jobs: ['5908250']
2021-02-25T12:30:55.995 INFO:scrape:suites: ['centos_latest', 'clusters/9-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'multimds/verify/{begin', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{fuse-default-perm-no', 'session_timeout', 'tasks/cfuse_workunit_suites_dbench', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}']
2021-02-25T12:30:55.995 INFO:scrape:
2021-02-25T12:30:55.996 INFO:scrape:Failure: "2021-02-25 08:27:54.356906 mon.a (mon.0) 214 : cluster [WRN] Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)" in cluster log
2021-02-25T12:30:55.996 INFO:scrape:1 jobs: ['5908227']
2021-02-25T12:30:55.997 INFO:scrape:suites: ['begin', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/yes', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_16.04}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/rhel/{k-distro', 'q_check_counter/check_counter', 'rhel_latest}', 'session_timeout', 'tasks/cfuse_workunit_suites_ffsb}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-02-25T12:30:55.997 INFO:scrape:
2021-02-25T12:30:55.998 INFO:scrape:Failure: 
2021-02-25T12:30:55.998 INFO:scrape:1 jobs: ['5908234']
2021-02-25T12:30:55.999 INFO:scrape:suites: ['centos_latest', 'clusters/9-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'multimds/verify/{begin', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{fuse-default-perm-no', 'session_timeout', 'tasks/cfuse_workunit_suites_fsstress', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}']
2021-02-25T12:30:55.999 INFO:scrape:
2021-02-25T12:30:55.999 INFO:scrape:Failure: Command failed (workunit test suites/fsstress.sh) on smithi064 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ec6639ba67b1c03b7f7d45e9d850924618ce9b4d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'
2021-02-25T12:30:56.000 INFO:scrape:1 jobs: ['5908240']
2021-02-25T12:30:56.000 INFO:scrape:suites: ['begin', 'ceph-thrash/default', 'clusters/3-mds-2-standby', 'conf/{client', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multimds/thrash/{0-supported-random-distro$/{centos_latest}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/rhel/{k-distro', 'overrides/{fuse-default-perm-no', 'rhel_latest}', 'session_timeout', 'tasks/cfuse_workunit_suites_fsstress}', 'thrash/{frag_enable', 'thrash_debug}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-02-25T12:30:56.001 INFO:scrape:
2021-02-25T12:30:56.002 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/
failed:  4
dead:    0
running: 0
waiting: 0
queued:  0
passed:  20


Fail
=================================================================
[5908227]  multimds/basic/{0-supported-random-distro$/{ubuntu_16.04} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
-----------------------------------------------------------------
time:   00:46:12
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908227/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908227/

    "2021-02-25 08:27:54.356906 mon.a (mon.0) 214 : cluster [WRN] Health check
    failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)" in cluster
    log

[5908234]  multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind}
-----------------------------------------------------------------
time:   04:07:23
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908234/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908234/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e1447a78f8a84e919e09c3f5133471df

[5908240]  multimds/thrash/{0-supported-random-distro$/{centos_latest} begin ceph-thrash/default clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:16:54
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908240/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908240/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c036c62cd7d54b77aec1a3ebec8af20f

    Command failed (workunit test suites/fsstress.sh) on smithi064 with status
    2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=ec6639ba67b1c03b7f7d45e9d850924618ce9b4d
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[5908250]  multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:58:45
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908250/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908250/

    reached maximum tries (50) after waiting for 300 seconds



Pass
=================================================================
[5908226] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   01:03:45
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908226/

[5908230] multimds/basic/{0-supported-random-distro$/{ubuntu_16.04} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
time:   00:15:45
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908230/

[5908232] multimds/thrash/{0-supported-random-distro$/{ubuntu_16.04} begin ceph-thrash/default clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:10:50
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908232/

[5908236] multimds/basic/{0-supported-random-distro$/{ubuntu_16.04} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:19:05
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908236/

[5908238] multimds/basic/{0-supported-random-distro$/{ubuntu_16.04} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:11:25
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908238/

[5908242] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
time:   00:15:36
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908242/

[5908244] multimds/basic/{0-supported-random-distro$/{rhel_7} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
time:   00:14:47
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908244/

[5908245] multimds/basic/{0-supported-random-distro$/{rhel_7} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
time:   00:27:35
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908245/

[5908248] multimds/thrash/{0-supported-random-distro$/{ubuntu_16.04} begin ceph-thrash/default clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:10:22
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908248/

[5908252] multimds/basic/{0-supported-random-distro$/{rhel_7} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   00:43:10
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908252/

[5908254] multimds/basic/{0-supported-random-distro$/{rhel_7} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   01:01:33
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908254/

[5908256] multimds/thrash/{0-supported-random-distro$/{rhel_7} begin ceph-thrash/default clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:20:28
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908256/

[5908258] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
time:   00:16:57
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908258/

[5908260] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:14:27
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908260/

[5908262] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
time:   00:24:49
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908262/

[5908264] multimds/thrash/{0-supported-random-distro$/{centos_latest} begin ceph-thrash/default clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:14:46
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908264/

[5908265] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
time:   00:57:51
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908265/

[5908266] multimds/basic/{0-supported-random-distro$/{ubuntu_16.04} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   01:09:21
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908266/

[5908267] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   00:44:06
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908267/

[5908268] multimds/thrash/{0-supported-random-distro$/{centos_latest} begin ceph-thrash/default clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:19:40
info:   http://pulpito.ceph.com/yuriw-2021-02-23_19:25:06-multimds-wip-yuri2-testing-2021-02-23-0932-nautilus-distro-basic-smithi/5908268/