2021-11-27T23:44:25.223 INFO:root:teuthology version: 1.1.0-a0bc3188
2021-11-27T23:44:25.225 INFO:teuthology.results:subset: '6887/9999'
2021-11-27T23:44:25.225 INFO:teuthology.results:seed: '7124'
2021-11-28T23:58:23.492 INFO:root:teuthology version: 1.1.0-a0bc3188
2021-11-28T23:58:23.550 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-11-29T00:55:27.940 INFO:teuthology.results:Tests finished! gathering results...
2021-11-29T00:55:31.887 INFO:scrape:Found 24 jobs
2021-11-29T00:55:41.806 INFO:scrape:Found 3 distinct failure reasons
2021-11-29T00:55:41.808 INFO:scrape:Failure: The following counters failed to be set on mds daemons: {'mds.imported', 'mds.exported'}
2021-11-29T00:55:41.808 INFO:scrape:2 jobs: ['6530142', '6530163']
2021-11-29T00:55:41.808 INFO:scrape:suites intersection: ['begin', 'conf/{client', 'fuse-default-perm-no}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{basic/{frag_enable', 'q_check_counter/check_counter', 'session_timeout', 'tasks/cephfs_test_exports}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-11-29T00:55:41.809 INFO:scrape:suites union: ['begin', 'clusters/3-mds', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'inline/yes', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{centos_8}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_18.04}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'q_check_counter/check_counter', 'rhel_8}', 'session_timeout', 'tasks/cephfs_test_exports}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-11-29T00:55:41.809 INFO:scrape:
2021-11-29T00:55:41.810 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds
2021-11-29T00:55:41.810 INFO:scrape:1 jobs: ['6530145']
2021-11-29T00:55:41.811 INFO:scrape:suites: ['centos_latest', 'clusters/3-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'multimds/verify/{begin', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{fuse-default-perm-no', 'session_timeout', 'tasks/cfuse_workunit_suites_fsstress', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}']
2021-11-29T00:55:41.811 INFO:scrape:
2021-11-29T00:55:41.812 INFO:scrape:Dead: b'2021-11-28T16:03:31.622 DEBUG:teuthology.exit:Finished running handlers'
2021-11-29T00:55:41.812 INFO:scrape:1 jobs: ['6530151']
2021-11-29T00:55:41.813 INFO:scrape:suites: ['begin', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{centos_8}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'q_check_counter/check_counter', 'session_timeout', 'tasks/cfuse_workunit_suites_blogbench}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-11-29T00:55:41.813 INFO:scrape:
2021-11-29T00:55:41.815 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba
=================================================================
info:    http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/
failed:  3
dead:    1
running: 0
waiting: 0
queued:  0
passed:  20


Fail
=================================================================
[6530142]  multimds/basic/{0-supported-random-distro$/{ubuntu_18.04} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
-----------------------------------------------------------------
time:   00:35:27
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530142/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530142/

    The following counters failed to be set on mds daemons: {'mds.imported',
    'mds.exported'}

[6530145]  multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind}
-----------------------------------------------------------------
time:   01:49:49
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530145/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530145/

    reached maximum tries (50) after waiting for 300 seconds

[6530163]  multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
-----------------------------------------------------------------
time:   00:36:11
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530163/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530163/

    The following counters failed to be set on mds daemons: {'mds.exported',
    'mds.imported'}



Dead
=================================================================
[6530151]  multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530151/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530151/



Pass
=================================================================
[6530141] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   01:12:41
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530141/

[6530143] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
time:   00:21:34
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530143/

[6530144] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mon clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:12:21
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530144/

[6530146] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   01:33:14
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530146/

[6530147] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   00:26:51
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530147/

[6530148] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mds clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:11:04
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530148/

[6530149] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   01:01:47
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530149/

[6530150] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:11:38
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530150/

[6530152] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mon clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:12:18
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530152/

[6530153] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind}
time:   01:26:56
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530153/

[6530154] multimds/basic/{0-supported-random-distro$/{ubuntu_18.04} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   01:00:03
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530154/

[6530155] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   00:41:33
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530155/

[6530156] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   01:08:59
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530156/

[6530157] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:11:39
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530157/

[6530158] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
time:   00:12:35
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530158/

[6530159] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:20:54
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530159/

[6530160] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind}
time:   01:45:53
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530160/

[6530161] multimds/thrash/{0-supported-random-distro$/{ubuntu_18.04} begin ceph-thrash/mon clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/none objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:11:55
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530161/

[6530162] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:10:41
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530162/

[6530164] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   01:00:20
info:   http://pulpito.ceph.com/teuthology-2021-11-27_13:15:03-multimds-octopus-distro-default-gibba/6530164/