2022-06-18T13:15:12.030 INFO:root:teuthology version: 1.1.1.dev374+g31a356e0
2022-06-18T13:15:12.031 INFO:teuthology.results:subset: '4569/9999'
2022-06-18T13:15:12.032 INFO:teuthology.results:seed: '2781'
2022-06-18T13:15:41.061 INFO:root:teuthology version: 1.1.1.dev374+g31a356e0
2022-06-18T13:15:41.063 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-06-18T14:28:46.613 INFO:teuthology.results:Tests finished! gathering results...
2022-06-18T14:28:50.092 INFO:scrape:Found 24 jobs
2022-06-18T14:28:57.137 INFO:scrape:Found 3 distinct failure reasons
2022-06-18T14:28:57.139 INFO:scrape:Failure: module 'teuthology.misc' has no attribute 'get_valgrind_args'
2022-06-18T14:28:57.139 INFO:scrape:3 jobs: ['6885157', '6885165', '6885173']
2022-06-18T14:28:57.140 INFO:scrape:suites intersection: ['centos_latest', 'conf/{client', 'mds', 'mon', 'mon-debug', 'multimds/verify/{begin', 'osd}', 'overrides/{fuse-default-perm-no', 'session_timeout', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}']
2022-06-18T14:28:57.140 INFO:scrape:suites union: ['centos_latest', 'clusters/3-mds', 'clusters/9-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'multimds/verify/{begin', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{fuse-default-perm-no', 'session_timeout', 'tasks/cfuse_workunit_suites_dbench', 'tasks/cfuse_workunit_suites_fsstress', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}']
2022-06-18T14:28:57.141 INFO:scrape:
2022-06-18T14:28:57.141 INFO:scrape:Failure: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}
2022-06-18T14:28:57.142 INFO:scrape:1 jobs: ['6885176']
2022-06-18T14:28:57.142 INFO:scrape:suites: ['begin', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/yes', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_18.04}', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'q_check_counter/check_counter', 'session_timeout', 'tasks/cephfs_test_exports}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-06-18T14:28:57.143 INFO:scrape:
2022-06-18T14:28:57.143 INFO:scrape:Crash: Cannot connect to remote host smithi146
2022-06-18T14:28:57.144 INFO:scrape:ceph version 15.2.16-260-gb3641a55 (b3641a55c6b9ed8f813eab4ef0cde6337b2b7ee6) octopus (stable)
 1: (()+0x14420) [0x7fc8db87b420]
 2: (gsignal()+0xcb) [0x7fc8db36400b]
 3: (abort()+0x12b) [0x7fc8db343859]
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7fc8dbcfabd8]
 5: (()+0x26cd85) [0x7fc8dbcfad85]
 6: (MDSTableClient::handle_request(boost::intrusive_ptr<MMDSTableRequest const> const&)+0x1554) [0x561eb0d45a64]
 7: (MDSRank::handle_message(boost::intrusive_ptr<Message const> const&)+0xeb5) [0x561eb0a60f45]
 8: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x68b) [0x561eb0a7497b]
 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0xca) [0x561eb0a750aa]
 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x128) [0x561eb0a52a98]
 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x460) [0x7fc8dbf51830]
 12: (DispatchQueue::entry()+0x58f) [0x7fc8dbf4f0cf]
 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc8dc006941]
 14: (()+0x8609) [0x7fc8db86f609]
 15: (clone()+0x43) [0x7fc8db440133]
2022-06-18T14:28:57.144 INFO:scrape:1 jobs: ['6885178']
2022-06-18T14:28:57.145 INFO:scrape:suites: ['begin', 'clusters/3-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_18.04}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'q_check_counter/check_counter', 'session_timeout', 'tasks/cephfs_test_snapshots}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-06-18T14:28:57.145 INFO:scrape:
2022-06-18T14:28:57.147 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/
failed:  5
dead:    0
running: 0
waiting: 0
queued:  0
passed:  19


Fail
=================================================================
[6885157]  multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:22:02
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885157/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885157/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4cd848833adc4f74bc80d31dd31d1b26

    module 'teuthology.misc' has no attribute 'get_valgrind_args'

[6885165]  multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind}
-----------------------------------------------------------------
time:   00:30:53
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885165/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885165/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a1ea28abfa044591965fb9deadfc9167

    module 'teuthology.misc' has no attribute 'get_valgrind_args'

[6885173]  multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind}
-----------------------------------------------------------------
time:   00:29:56
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885173/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885173/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6366f3d5199a4f4f99a90003880cb706

    module 'teuthology.misc' has no attribute 'get_valgrind_args'

[6885176]  multimds/basic/{0-supported-random-distro$/{ubuntu_18.04} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
-----------------------------------------------------------------
time:   00:48:34
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885176/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885176/

    The following counters failed to be set on mds daemons: {'mds.exported',
    'mds.imported'}

[6885178]  multimds/basic/{0-supported-random-distro$/{ubuntu_18.04} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
-----------------------------------------------------------------
time:   00:28:46
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885178/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885178/

    Cannot connect to remote host smithi146



Pass
=================================================================
[6885156] multimds/basic/{0-supported-random-distro$/{ubuntu_18.04} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
time:   00:29:38
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885156/

[6885158] multimds/basic/{0-supported-random-distro$/{ubuntu_18.04} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   00:43:38
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885158/

[6885159] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:18:17
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885159/

[6885160] multimds/basic/{0-supported-random-distro$/{ubuntu_18.04} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   01:03:56
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885160/

[6885161] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   01:02:29
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885161/

[6885162] multimds/basic/{0-supported-random-distro$/{ubuntu_18.04} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:20:40
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885162/

[6885163] multimds/thrash/{0-supported-random-distro$/{ubuntu_18.04} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:25:23
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885163/

[6885164] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
time:   00:30:53
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885164/

[6885166] multimds/basic/{0-supported-random-distro$/{ubuntu_18.04} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   00:51:56
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885166/

[6885167] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:18:35
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885167/

[6885168] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   00:36:26
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885168/

[6885169] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   01:03:16
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885169/

[6885170] multimds/basic/{0-supported-random-distro$/{ubuntu_18.04} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
time:   00:25:37
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885170/

[6885171] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:26:03
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885171/

[6885172] multimds/basic/{0-supported-random-distro$/{ubuntu_18.04} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:33:09
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885172/

[6885174] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:18:47
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885174/

[6885175] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mon clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:17:59
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885175/

[6885177] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   00:58:47
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885177/

[6885179] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:24:48
info:   http://pulpito.ceph.com/teuthology-2022-06-18_13:15:02-multimds-octopus-distro-default-smithi/6885179/