2022-07-05T11:59:58.551 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3 2022-07-05T11:59:58.552 INFO:teuthology.results:subset: '4345/9999' 2022-07-05T11:59:58.552 INFO:teuthology.results:seed: '6778' 2022-07-05T12:32:27.199 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3 2022-07-05T12:32:27.227 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-07-05T13:48:34.150 INFO:teuthology.results:Tests finished! gathering results... 2022-07-05T13:48:36.820 INFO:scrape:Found 24 jobs 2022-07-05T13:48:40.108 INFO:scrape:Found 2 distinct failure reasons 2022-07-05T13:48:40.110 INFO:scrape:Crash: Cannot connect to remote host smithi137 2022-07-05T13:48:40.110 INFO:scrape:ceph version 15.2.16-305-gf90f9b73 (f90f9b732959620c26243b10b198d421ece2e917) octopus (stable) 1: (()+0x12ce0) [0x7fd5e0be3ce0] 2: (gsignal()+0x10f) [0x7fd5df640a9f] 3: (abort()+0x127) [0x7fd5df613e05] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fd5e2002c53] 5: (()+0x27de1c) [0x7fd5e2002e1c] 6: (MDSTableClient::handle_request(boost::intrusive_ptr const&)+0x128b) [0x55684dde214b] 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xddc) [0x55684db55c4c] 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x7bb) [0x55684db57e6b] 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x47) [0x55684db58447] 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x128) [0x55684db45638] 11: (DispatchQueue::entry()+0x126a) [0x7fd5e2237e7a] 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd5e22d7a31] 13: (()+0x81ca) [0x7fd5e0bd91ca] 14: (clone()+0x43) [0x7fd5df62bdd3] 2022-07-05T13:48:40.111 INFO:scrape:1 jobs: ['6910375'] 2022-07-05T13:48:40.111 INFO:scrape:suites: ['begin', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/yes', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{centos_8}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/testing/{flavor/centos_latest', 'q_check_counter/check_counter', 'session_timeout', 'tasks/cephfs_test_snapshots}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-05T13:48:40.112 INFO:scrape: 2022-07-05T13:48:40.113 INFO:scrape:Failure: The following counters failed to be set on mds daemons: {'mds.imported', 'mds.exported'} 2022-07-05T13:48:40.113 INFO:scrape:1 jobs: ['6910373'] 2022-07-05T13:48:40.114 INFO:scrape:suites: ['begin', 'clusters/3-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/no', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_18.04}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/stock/{k-stock', 'q_check_counter/check_counter', 'rhel_8}', 'session_timeout', 'tasks/cephfs_test_exports}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-07-05T13:48:40.114 INFO:scrape: 2022-07-05T13:48:40.116 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 22 Fail ================================================================= [6910373] multimds/basic/{0-supported-random-distro$/{ubuntu_18.04} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports} ----------------------------------------------------------------- time: 00:41:38 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910373/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910373/ The following counters failed to be set on mds daemons: {'mds.imported', 'mds.exported'} [6910375] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots} ----------------------------------------------------------------- time: 00:25:31 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910375/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910375/ Cannot connect to remote host smithi137 Pass ================================================================= [6910357] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc} time: 00:49:52 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910357/ [6910358] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep} time: 00:29:57 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910358/ [6910359] multimds/basic/{0-supported-random-distro$/{ubuntu_18.04} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats} time: 00:13:51 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910359/ [6910360] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mon clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress} time: 00:17:52 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910360/ [6910361] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench} time: 00:37:10 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910361/ [6910362] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind} time: 00:51:48 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910362/ [6910363] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench} time: 00:56:51 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910363/ [6910364] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/mds clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd} time: 00:12:47 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910364/ [6910365] multimds/basic/{0-supported-random-distro$/{ubuntu_18.04} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb} time: 00:39:02 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910365/ [6910366] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep} time: 00:18:41 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910366/ [6910367] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress} time: 00:18:18 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910367/ [6910368] multimds/thrash/{0-supported-random-distro$/{centos_8} begin ceph-thrash/mon clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress} time: 00:17:31 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910368/ [6910369] multimds/basic/{0-supported-random-distro$/{centos_8} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx} time: 00:22:58 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910369/ [6910370] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind} time: 00:56:07 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910370/ [6910371] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd} time: 00:15:40 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910371/ [6910372] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mds clusters/3-mds-2-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd} time: 00:16:28 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910372/ [6910374] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep} time: 00:18:05 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910374/ [6910376] multimds/thrash/{0-supported-random-distro$/{rhel_8} begin ceph-thrash/mon clusters/9-mds-3-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress} time: 00:18:24 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910376/ [6910377] multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build} time: 01:08:41 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910377/ [6910378] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind} time: 00:52:03 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910378/ [6910379] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc} time: 01:04:45 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910379/ [6910380] multimds/thrash/{0-supported-random-distro$/{ubuntu_18.04} begin ceph-thrash/mds clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd} time: 00:13:06 info: http://pulpito.ceph.com/teuthology-2022-07-02_13:15:03-multimds-octopus-distro-default-smithi/6910380/