2021-12-13T21:16:06.104 INFO:root:teuthology version: 1.1.0-95a7d479
2021-12-13T21:16:06.106 INFO:teuthology.results:subset: '111/120000'
2021-12-13T21:16:06.106 INFO:teuthology.results:seed: '2859'
2021-12-13T21:36:08.791 INFO:root:teuthology version: 1.1.0-95a7d479
2021-12-13T21:36:08.824 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-12-14T00:31:21.737 INFO:teuthology.results:Tests finished! gathering results...
2021-12-14T00:31:23.902 INFO:scrape:Found 24 jobs
2021-12-14T00:31:27.013 INFO:scrape:Found 2 distinct failure reasons
2021-12-14T00:31:27.014 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds
2021-12-14T00:31:27.014 INFO:scrape:2 jobs: ['6561031', '6561023']
2021-12-14T00:31:27.015 INFO:scrape:suites intersection: ['centos_latest', 'clusters/9-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'multimds/verify/{begin', 'osd}', 'overrides/{fuse-default-perm-no', 'session_timeout', 'tasks/cfuse_workunit_suites_dbench', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}']
2021-12-14T00:31:27.015 INFO:scrape:suites union: ['centos_latest', 'clusters/9-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'multimds/verify/{begin', 'objectstore-ec/bluestore-comp', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{fuse-default-perm-no', 'session_timeout', 'tasks/cfuse_workunit_suites_dbench', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}']
2021-12-14T00:31:27.016 INFO:scrape:
2021-12-14T00:31:27.016 INFO:scrape:Failure: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}
2021-12-14T00:31:27.017 INFO:scrape:1 jobs: ['6561032']
2021-12-14T00:31:27.017 INFO:scrape:suites: ['begin', 'clusters/3-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/yes', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{centos_8}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'q_check_counter/check_counter', 'session_timeout', 'tasks/cephfs_test_exports}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-12-14T00:31:27.017 INFO:scrape:
2021-12-14T00:31:27.019 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/
failed:  3
dead:    0
running: 0
waiting: 0
queued:  0
passed:  21


Fail
=================================================================
[6561023]  multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
-----------------------------------------------------------------
time:   01:03:07
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561023/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561023/

    reached maximum tries (50) after waiting for 300 seconds

[6561031]  multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:59:32
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561031/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561031/

    reached maximum tries (50) after waiting for 300 seconds

[6561032]  multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
-----------------------------------------------------------------
time:   00:41:56
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561032/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561032/

    The following counters failed to be set on mds daemons: {'mds.exported',
    'mds.imported'}



Pass
=================================================================
[6561018] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:14:14
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561018/

[6561019] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
time:   00:19:41
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561019/

[6561020] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
time:   00:27:42
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561020/

[6561021] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mds clusters/3-mds-2-standby conf/{client mds mon osd} mount/fuse msgr-failures/none objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:12:21
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561021/

[6561022] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   00:54:50
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561022/

[6561024] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   00:31:59
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561024/

[6561025] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:19:16
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561025/

[6561026] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
time:   00:17:53
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561026/

[6561027] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
time:   00:17:36
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561027/

[6561028] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:18:35
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561028/

[6561029] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mds clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:15:25
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561029/

[6561030] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:15:16
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561030/

[6561033] multimds/thrash/{0-supported-random-distro$/{ubuntu_18.04} begin ceph-thrash/mon clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:17:53
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561033/

[6561034] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
time:   00:23:44
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561034/

[6561035] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   01:02:35
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561035/

[6561036] multimds/basic/{0-supported-random-distro$/{ubuntu_18.04} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   00:49:59
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561036/

[6561037] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/3-mds-2-standby conf/{client mds mon osd} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:11:10
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561037/

[6561038] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   00:45:46
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561038/

[6561039] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind}
time:   02:50:32
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561039/

[6561040] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:13:15
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561040/

[6561041] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mon clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:20:14
info:   http://pulpito.ceph.com/yuriw-2021-12-13_20:15:12-multimds-wip-yuri5-testing-2021-12-13-0836-octopus-distro-default-smithi/6561041/