2021-12-04T13:15:42.261 INFO:root:teuthology version: 1.1.0-844d1377
2021-12-04T13:15:42.264 INFO:teuthology.results:subset: '3604/9999'
2021-12-04T13:15:42.264 INFO:teuthology.results:seed: '3869'
2021-12-04T13:15:54.357 INFO:root:teuthology version: 1.1.0-844d1377
2021-12-04T13:15:54.358 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-12-05T01:16:47.066 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2021-12-05T01:16:49.513 INFO:scrape:Found 24 jobs
2021-12-05T01:16:54.017 INFO:scrape:Found 3 distinct failure reasons
2021-12-05T01:16:54.018 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds
2021-12-05T01:16:54.018 INFO:scrape:1 jobs: ['6544504']
2021-12-05T01:16:54.019 INFO:scrape:suites: ['centos_latest', 'clusters/9-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'multimds/verify/{begin', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{fuse-default-perm-no', 'session_timeout', 'tasks/cfuse_workunit_suites_dbench', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}']
2021-12-05T01:16:54.019 INFO:scrape:
2021-12-05T01:16:54.020 INFO:scrape:Dead: b'2021-12-05T01:16:46.595 DEBUG:teuthology.orchestra.run.smithi198:> sudo logrotate /etc/logrotate.d/ceph-test.conf'
2021-12-05T01:16:54.020 INFO:scrape:2 jobs: ['6544515', '6544509']
2021-12-05T01:16:54.021 INFO:scrape:suites intersection: ['begin', 'conf/{client', 'fuse-default-perm-no}', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_latest}', 'osd}', 'overrides/{basic/{frag_enable', 'q_check_counter/check_counter', 'session_timeout', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-12-05T01:16:54.021 INFO:scrape:suites union: ['begin', 'clusters/3-mds', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'inline/yes', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_latest}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'q_check_counter/check_counter', 'session_timeout', 'tasks/cfuse_workunit_kernel_untar_build}', 'tasks/cfuse_workunit_suites_blogbench}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-12-05T01:16:54.021 INFO:scrape:
2021-12-05T01:16:54.022 INFO:scrape:Failure: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}
2021-12-05T01:16:54.022 INFO:scrape:2 jobs: ['6544505', '6544527']
2021-12-05T01:16:54.023 INFO:scrape:suites intersection: ['begin', 'conf/{client', 'fuse-default-perm-no}', 'mds', 'mon', 'osd}', 'overrides/{basic/{frag_enable', 'q_check_counter/check_counter', 'session_timeout', 'tasks/cephfs_test_exports}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-12-05T01:16:54.023 INFO:scrape:suites union: ['begin', 'clusters/3-mds', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'inline/yes', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{rhel_8}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_latest}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/stock/{k-stock', 'q_check_counter/check_counter', 'rhel_8}', 'session_timeout', 'tasks/cephfs_test_exports}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-12-05T01:16:54.024 INFO:scrape:
2021-12-05T01:16:54.025 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/
failed:  3
dead:    0
running: 2
waiting: 0
queued:  0
passed:  19


Fail
=================================================================
[6544504]  multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
-----------------------------------------------------------------
time:   01:07:55
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544504/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544504/

    reached maximum tries (50) after waiting for 300 seconds

[6544505]  multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
-----------------------------------------------------------------
time:   00:38:59
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544505/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544505/

    The following counters failed to be set on mds daemons: {'mds.exported',
    'mds.imported'}

[6544527]  multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
-----------------------------------------------------------------
time:   00:42:23
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544527/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544527/

    The following counters failed to be set on mds daemons: {'mds.imported',
    'mds.exported'}



Running
=================================================================
[6544509] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544509/

[6544515] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544515/



Pass
=================================================================
[6544506] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:23:33
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544506/

[6544507] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
time:   00:28:15
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544507/

[6544508] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   01:02:12
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544508/

[6544510] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:14:49
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544510/

[6544511] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   01:56:23
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544511/

[6544512] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind}
time:   03:06:07
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544512/

[6544513] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:16:02
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544513/

[6544514] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:20:49
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544514/

[6544516] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   00:58:26
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544516/

[6544517] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   00:51:19
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544517/

[6544518] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:15:19
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544518/

[6544519] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   00:28:18
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544519/

[6544520] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind}
time:   02:32:06
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544520/

[6544521] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
time:   00:16:06
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544521/

[6544522] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:21:00
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544522/

[6544523] multimds/basic/{0-supported-random-distro$/{ubuntu_18.04} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:34:12
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544523/

[6544524] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   00:59:17
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544524/

[6544525] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:13:07
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544525/

[6544526] multimds/thrash/{0-supported-random-distro$/{ubuntu_18.04} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:13:27
info:   http://pulpito.ceph.com/teuthology-2021-12-04_13:15:03-multimds-octopus-distro-default-smithi/6544526/