2022-05-21T13:38:48.707 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01
2022-05-21T13:38:48.709 INFO:teuthology.results:subset: '4076/9999'
2022-05-21T13:38:48.709 INFO:teuthology.results:seed: '2086'
2022-05-21T21:24:22.604 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01
2022-05-21T21:24:22.630 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-05-22T03:40:50.981 INFO:teuthology.results:Tests finished! gathering results...
2022-05-22T03:40:54.223 INFO:scrape:Found 24 jobs
2022-05-22T03:40:58.909 INFO:scrape:Found 4 distinct failure reasons
2022-05-22T03:40:58.911 INFO:scrape:Crash: Cannot connect to remote host smithi131
2022-05-22T03:40:58.911 INFO:scrape:ceph version 15.2.16-214-gffda8f3f (ffda8f3f585d85c3f2add12006650097fc8268ae) octopus (stable)
 1: (()+0x12ce0) [0x7f5d02813ce0]
 2: (gsignal()+0x10f) [0x7f5d01271a4f]
 3: (abort()+0x127) [0x7f5d01244db5]
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f5d03c32c53]
 5: (()+0x27de1c) [0x7f5d03c32e1c]
 6: (MDSTableClient::handle_request(boost::intrusive_ptr<MMDSTableRequest const> const&)+0x128b) [0x5594b9784fbb]
 7: (MDSRank::handle_message(boost::intrusive_ptr<Message const> const&)+0xddc) [0x5594b94f8c5c]
 8: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x7bb) [0x5594b94fae7b]
 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x47) [0x5594b94fb457]
 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x128) [0x5594b94e8648]
 11: (DispatchQueue::entry()+0x126a) [0x7f5d03e67dfa]
 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5d03f079b1]
 13: (()+0x81ca) [0x7f5d028091ca]
 14: (clone()+0x43) [0x7f5d0125cd83]
2022-05-22T03:40:58.912 INFO:scrape:1 jobs: ['6843200']
2022-05-22T03:40:58.912 INFO:scrape:suites: ['begin', 'clusters/3-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_18.04}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'q_check_counter/check_counter', 'session_timeout', 'tasks/cephfs_test_snapshots}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-22T03:40:58.912 INFO:scrape:
2022-05-22T03:40:58.913 INFO:scrape:Failure: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}
2022-05-22T03:40:58.913 INFO:scrape:1 jobs: ['6843198']
2022-05-22T03:40:58.914 INFO:scrape:suites: ['begin', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/yes', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{centos_8}', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/stock/{k-stock', 'q_check_counter/check_counter', 'rhel_8}', 'session_timeout', 'tasks/cephfs_test_exports}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-22T03:40:58.914 INFO:scrape:
2022-05-22T03:40:58.915 INFO:scrape:Dead: b'2022-05-22T03:37:41.572 DEBUG:teuthology.exit:Finished running handlers'
2022-05-22T03:40:58.915 INFO:scrape:1 jobs: ['6843203']
2022-05-22T03:40:58.916 INFO:scrape:suites: ['centos_latest', 'clusters/3-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'multimds/verify/{begin', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{fuse-default-perm-no', 'session_timeout', 'tasks/cfuse_workunit_suites_fsstress', 'validater/lockdep}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}']
2022-05-22T03:40:58.916 INFO:scrape:
2022-05-22T03:40:58.916 INFO:scrape:Failure: SSH connection to smithi131 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1'
2022-05-22T03:40:58.917 INFO:scrape:1 jobs: ['6843192']
2022-05-22T03:40:58.917 INFO:scrape:suites: ['begin', 'clusters/3-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{rhel_8}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'q_check_counter/check_counter', 'session_timeout', 'tasks/cfuse_workunit_suites_fsstress}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2022-05-22T03:40:58.918 INFO:scrape:
2022-05-22T03:40:58.927 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/
failed:  3
dead:    1
running: 0
waiting: 0
queued:  0
passed:  20


Fail
=================================================================
[6843192]  multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:34:04
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843192/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843192/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8b87cd5147c34d8ab242a3943a923cdb

    SSH connection to smithi131 was lost: 'sudo rm -rf --
    /home/ubuntu/cephtest/workunits.list.client.1
    /home/ubuntu/cephtest/clone.client.1'

[6843198]  multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
-----------------------------------------------------------------
time:   00:43:13
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843198/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843198/

    The following counters failed to be set on mds daemons: {'mds.exported',
    'mds.imported'}

[6843200]  multimds/basic/{0-supported-random-distro$/{ubuntu_18.04} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
-----------------------------------------------------------------
time:   00:28:31
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843200/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843200/

    Cannot connect to remote host smithi131



Dead
=================================================================
[6843203]  multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843203/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843203/

    hit max job timeout



Pass
=================================================================
[6843190] multimds/basic/{0-supported-random-distro$/{ubuntu_18.04} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   00:34:30
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843190/

[6843191] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
time:   00:54:06
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843191/

[6843193] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/3-mds-2-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:11:50
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843193/

[6843194] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:28:15
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843194/

[6843195] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
time:   00:22:26
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843195/

[6843196] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:16:31
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843196/

[6843197] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mon clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:17:30
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843197/

[6843199] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
time:   00:54:42
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843199/

[6843201] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:18:33
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843201/

[6843202] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   01:09:27
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843202/

[6843204] multimds/basic/{0-supported-random-distro$/{ubuntu_18.04} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   00:45:46
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843204/

[6843205] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:19:52
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843205/

[6843206] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:18:15
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843206/

[6843207] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
time:   00:54:12
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843207/

[6843208] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
time:   00:28:43
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843208/

[6843209] multimds/thrash/{0-supported-random-distro$/{ubuntu_18.04} begin ceph-thrash/mds clusters/3-mds-2-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:11:52
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843209/

[6843210] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   00:57:10
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843210/

[6843211] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   00:56:35
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843211/

[6843212] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   00:44:34
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843212/

[6843213] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mon clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:17:25
info:   http://pulpito.ceph.com/teuthology-2022-05-21_13:15:03-multimds-octopus-distro-default-smithi/6843213/