2020-11-06T06:46:07.956 INFO:root:teuthology version: 1.0.0-41b569f
2020-11-06T06:46:07.957 INFO:teuthology.results:subset: '6/2999'
2020-11-06T06:46:07.957 INFO:teuthology.results:seed: '704'
2020-11-06T10:45:37.810 INFO:root:teuthology version: 1.0.0-41b569f
2020-11-06T10:45:37.853 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-11-06T14:03:41.328 INFO:teuthology.results:Tests finished! gathering results...
2020-11-06T14:03:44.399 INFO:scrape:Found 24 jobs
2020-11-06T14:03:45.683 INFO:scrape:Found 3 distinct failure reasons
2020-11-06T14:03:45.684 INFO:scrape:Crash: "2020-11-06 07:19:10.883041 mon.a (mon.0) 248 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log
2020-11-06T14:03:45.684 INFO:scrape:ceph version 14.2.12-51-g15a2d601f0 (15a2d601f00ea1f10e59717244e34601f32b1a50) nautilus (stable)
 1: (()+0xf630) [0x7fd5f75c6630]
 2: (gsignal()+0x37) [0x7fd5f61a4387]
 3: (abort()+0x148) [0x7fd5f61a5a78]
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x7fd5f9708b76]
 5: (()+0x25ccef) [0x7fd5f9708cef]
 6: (ProtocolV1::handle_message_footer(char*, int)+0xf3d) [0x7fd5f99f48ed]
 7: (()+0x5410ed) [0x7fd5f99ed0ed]
 8: (AsyncConnection::process()+0x186) [0x7fd5f99decf6]
 9: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa15) [0x7fd5f9a2e115]
 10: (()+0x588a77) [0x7fd5f9a34a77]
 11: (()+0x827b9f) [0x7fd5f9cd3b9f]
 12: (()+0x7ea5) [0x7fd5f75beea5]
 13: (clone()+0x6d) [0x7fd5f626c8dd]
2020-11-06T14:03:45.685 INFO:scrape:1 jobs: ['5580866']
2020-11-06T14:03:45.685 INFO:scrape:suites: ['begin', 'ceph-thrash/default', 'clusters/3-mds-2-standby', 'conf/{client', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multimds/thrash/{0-supported-random-distro$/{ubuntu_16.04}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/rhel/{k-distro', 'overrides/{fuse-default-perm-no', 'rhel_latest}', 'tasks/cfuse_workunit_suites_fsstress}', 'thrash/{frag_enable', 'thrash_debug}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2020-11-06T14:03:45.686 INFO:scrape:
2020-11-06T14:03:45.686 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds
2020-11-06T14:03:45.686 INFO:scrape:1 jobs: ['5580860']
2020-11-06T14:03:45.686 INFO:scrape:suites: ['centos_latest', 'clusters/9-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'multimds/verify/{begin', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{fuse-default-perm-no', 'tasks/cfuse_workunit_suites_dbench', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}']
2020-11-06T14:03:45.687 INFO:scrape:
2020-11-06T14:03:45.687 INFO:scrape:Failure: "2020-11-06 07:22:57.695740 mon.a (mon.0) 221 : cluster [WRN] Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)" in cluster log
2020-11-06T14:03:45.687 INFO:scrape:1 jobs: ['5580854']
2020-11-06T14:03:45.688 INFO:scrape:suites: ['begin', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/yes', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_latest}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/rhel/{k-distro', 'q_check_counter/check_counter', 'rhel_latest}', 'tasks/cfuse_workunit_suites_ffsb}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2020-11-06T14:03:45.688 INFO:scrape:
2020-11-06T14:03:45.690 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/
failed:  3
dead:    0
running: 0
waiting: 0
queued:  0
passed:  21


Fail
=================================================================
[5580854]  multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
-----------------------------------------------------------------
time:   01:52:55
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580854/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580854/

    "2020-11-06 07:22:57.695740 mon.a (mon.0) 221 : cluster [WRN] Health check
    failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)" in cluster
    log

[5580860]  multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
-----------------------------------------------------------------
time:   01:00:31
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580860/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580860/

    reached maximum tries (50) after waiting for 300 seconds

[5580866]  multimds/thrash/{0-supported-random-distro$/{ubuntu_16.04} begin ceph-thrash/default clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:27:29
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580866/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580866/

    "2020-11-06 07:19:10.883041 mon.a (mon.0) 248 : cluster [WRN] Health check
    failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log



Pass
=================================================================
[5580848] multimds/basic/{0-supported-random-distro$/{ubuntu_16.04} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   01:01:10
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580848/

[5580850] multimds/thrash/{0-supported-random-distro$/{ubuntu_16.04} begin ceph-thrash/default clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:24:43
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580850/

[5580852] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
time:   00:23:13
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580852/

[5580856] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
time:   00:18:10
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580856/

[5580858] multimds/thrash/{0-supported-random-distro$/{rhel_latest} begin ceph-thrash/default clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:16:44
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580858/

[5580862] multimds/basic/{0-supported-random-distro$/{ubuntu_16.04} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:25:31
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580862/

[5580864] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:13:26
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580864/

[5580868] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
time:   00:21:29
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580868/

[5580870] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
time:   00:14:47
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580870/

[5580872] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
time:   00:27:26
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580872/

[5580874] multimds/thrash/{0-supported-random-distro$/{ubuntu_16.04} begin ceph-thrash/default clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/none objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:13:08
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580874/

[5580876] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
time:   00:58:20
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580876/

[5580878] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   00:41:08
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580878/

[5580880] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   01:14:45
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580880/

[5580882] multimds/thrash/{0-supported-random-distro$/{rhel_latest} begin ceph-thrash/default clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:20:47
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580882/

[5580884] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   01:08:39
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580884/

[5580886] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:20:00
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580886/

[5580888] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
time:   00:23:50
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580888/

[5580890] multimds/thrash/{0-supported-random-distro$/{ubuntu_16.04} begin ceph-thrash/default clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:16:06
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580890/

[5580892] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind}
time:   03:10:28
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580892/

[5580894] multimds/basic/{0-supported-random-distro$/{rhel_latest} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   01:11:12
info:   http://pulpito.ceph.com/teuthology-2020-11-01_13:15:03-multimds-nautilus-distro-basic-smithi/5580894/