2021-04-22T10:30:31.924 INFO:root:teuthology version: 1.1.0-2713a3cd
2021-04-22T10:30:31.925 INFO:teuthology.results:subset: '111/120000'
2021-04-22T10:30:31.925 INFO:teuthology.results:seed: '4114'
2021-04-22T10:52:49.131 INFO:root:teuthology version: 1.1.0-2713a3cd
2021-04-22T10:52:49.164 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-04-22T14:21:51.322 INFO:teuthology.results:Tests finished! gathering results...
2021-04-22T14:21:53.517 INFO:scrape:Found 24 jobs
2021-04-22T14:21:55.500 INFO:scrape:Found 4 distinct failure reasons
2021-04-22T14:21:55.501 INFO:scrape:Crash: SSH connection to smithi150 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0'
2021-04-22T14:21:55.501 INFO:scrape:ceph version 14.2.20-139-gb0d014fe259 (b0d014fe25986033c2db9422289a173f45eea553) nautilus (stable)
 1: (()+0xf630) [0x7f8ad6ec3630]
 2: (gsignal()+0x37) [0x7f8ad5aa03d7]
 3: (abort()+0x148) [0x7f8ad5aa1ac8]
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x7f8ad9006406]
 5: (()+0x25d57f) [0x7f8ad900657f]
 6: (ProtocolV1::handle_message_footer(char*, int)+0xf3d) [0x7f8ad92f675d]
 7: (()+0x545f5d) [0x7f8ad92eef5d]
 8: (AsyncConnection::process()+0x186) [0x7f8ad92e0b66]
 9: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa15) [0x7f8ad93301d5]
 10: (()+0x58db37) [0x7f8ad9336b37]
 11: (()+0x82d5ff) [0x7f8ad95d65ff]
 12: (()+0x7ea5) [0x7f8ad6ebbea5]
 13: (clone()+0x6d) [0x7f8ad5b689fd]
2021-04-22T14:21:55.501 INFO:scrape:1 jobs: ['6062481']
2021-04-22T14:21:55.501 INFO:scrape:suites: ['begin', 'ceph-thrash/default', 'clusters/3-mds-2-standby', 'conf/{client', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multimds/thrash/{0-supported-random-distro$/{centos_latest}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/rhel/{k-distro', 'overrides/{fuse-default-perm-no', 'rhel_latest}', 'session_timeout', 'tasks/cfuse_workunit_suites_fsstress}', 'thrash/{frag_enable', 'thrash_debug}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-04-22T14:21:55.501 INFO:scrape:
2021-04-22T14:21:55.502 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds
2021-04-22T14:21:55.502 INFO:scrape:1 jobs: ['6062486']
2021-04-22T14:21:55.502 INFO:scrape:suites: ['centos_latest', 'clusters/9-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'multimds/verify/{begin', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{fuse-default-perm-no', 'session_timeout', 'tasks/cfuse_workunit_suites_dbench', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}']
2021-04-22T14:21:55.502 INFO:scrape:
2021-04-22T14:21:55.502 INFO:scrape:Failure: 
2021-04-22T14:21:55.502 INFO:scrape:1 jobs: ['6062470']
2021-04-22T14:21:55.502 INFO:scrape:suites: ['centos_latest', 'clusters/9-mds', 'conf/{client', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'multimds/verify/{begin', 'objectstore-ec/filestore-xfs', 'osd}', 'overrides/{fuse-default-perm-no', 'session_timeout', 'tasks/cfuse_workunit_suites_fsstress', 'validater/valgrind}', 'verify/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}}']
2021-04-22T14:21:55.503 INFO:scrape:
2021-04-22T14:21:55.503 INFO:scrape:Failure: The following counters failed to be set on mds daemons: {'mds.imported', 'mds.exported'}
2021-04-22T14:21:55.503 INFO:scrape:1 jobs: ['6062475']
2021-04-22T14:21:55.503 INFO:scrape:suites: ['begin', 'clusters/9-mds', 'conf/{client', 'fuse-default-perm-no}', 'inline/yes', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'multimds/basic/{0-supported-random-distro$/{ubuntu_16.04}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{basic/{frag_enable', 'overrides/{distro/rhel/{k-distro', 'q_check_counter/check_counter', 'rhel_latest}', 'session_timeout', 'tasks/cephfs_test_exports}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-04-22T14:21:55.503 INFO:scrape:
2021-04-22T14:21:55.505 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/
failed:  4
dead:    0
running: 0
waiting: 0
queued:  0
passed:  20


Fail
=================================================================
[6062470]  multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/valgrind}
-----------------------------------------------------------------
time:   03:36:35
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062470/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062470/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b2e5b06b937846af80900a5e50efa781

[6062475]  multimds/basic/{0-supported-random-distro$/{ubuntu_16.04} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports}
-----------------------------------------------------------------
time:   00:15:04
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062475/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062475/

    The following counters failed to be set on mds daemons: {'mds.imported',
    'mds.exported'}

[6062481]  multimds/thrash/{0-supported-random-distro$/{centos_latest} begin ceph-thrash/default clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:20:07
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062481/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062481/
sentry: https://sentry.ceph.com/organizations/ceph/?query=82f42898f9e047bbab8119a108d66cae

    SSH connection to smithi150 was lost: 'sudo rm -rf --
    /home/ubuntu/cephtest/workunits.list.client.0
    /home/ubuntu/cephtest/clone.client.0'

[6062486]  multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
-----------------------------------------------------------------
time:   00:58:30
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062486/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062486/

    reached maximum tries (50) after waiting for 300 seconds



Pass
=================================================================
[6062466] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/lockdep}
time:   01:01:17
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062466/

[6062467] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   00:35:14
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062467/

[6062468] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsstress}
time:   00:20:45
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062468/

[6062469] multimds/thrash/{0-supported-random-distro$/{rhel_7} begin ceph-thrash/default clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:13:53
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062469/

[6062471] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_fsx}
time:   00:17:47
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062471/

[6062472] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_pjd}
time:   00:10:31
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062472/

[6062473] multimds/thrash/{0-supported-random-distro$/{ubuntu_latest} begin ceph-thrash/default clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:20:43
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062473/

[6062474] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
time:   00:15:38
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062474/

[6062476] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}
time:   00:30:31
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062476/

[6062477] multimds/thrash/{0-supported-random-distro$/{rhel_7} begin ceph-thrash/default clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:13:09
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062477/

[6062478] multimds/verify/{begin centos_latest clusters/9-mds conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_dbench validater/valgrind}
time:   00:56:39
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062478/

[6062479] multimds/basic/{0-supported-random-distro$/{ubuntu_16.04} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_kernel_untar_build}
time:   00:48:21
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062479/

[6062480] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/filestore-xfs overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_misc}
time:   00:42:17
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062480/

[6062482] multimds/verify/{begin centos_latest clusters/3-mds conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no verify/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down}} tasks/cfuse_workunit_suites_fsstress validater/lockdep}
time:   00:17:42
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062482/

[6062483] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_norstats}
time:   00:16:56
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062483/

[6062484] multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_blogbench}
time:   00:29:06
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062484/

[6062485] multimds/thrash/{0-supported-random-distro$/{centos_latest} begin ceph-thrash/default clusters/9-mds-3-standby conf/{client mds mon osd} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_pjd}
time:   00:13:10
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062485/

[6062487] multimds/basic/{0-supported-random-distro$/{ubuntu_latest} begin clusters/9-mds conf/{client mds mon osd} inline/yes mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_dbench}
time:   01:01:56
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062487/

[6062488] multimds/basic/{0-supported-random-distro$/{rhel_7} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-ec-root overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cfuse_workunit_suites_ffsb}
time:   00:43:21
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062488/

[6062489] multimds/thrash/{0-supported-random-distro$/{rhel_7} begin ceph-thrash/default clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
time:   00:20:42
info:   http://pulpito.ceph.com/yuriw-2021-04-21_16:22:49-multimds-wip-yuri2-testing-2021-04-20-0721-nautilus-distro-basic-smithi/6062489/