2021-03-02T01:45:01.224 INFO:root:teuthology version: 1.1.0-3082387b
2021-03-02T01:45:01.228 INFO:teuthology.results:seed: '1280'
2021-03-02T01:51:07.280 INFO:root:teuthology version: 1.1.0-3082387b
2021-03-02T01:51:07.480 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-03-02T04:29:19.490 INFO:teuthology.results:Tests finished! gathering results...
2021-03-02T04:29:26.733 INFO:scrape:Found 10 jobs
2021-03-02T04:29:39.299 INFO:scrape:Found 4 distinct failure reasons
2021-03-02T04:29:39.301 INFO:scrape:Crash: Found coredumps on ubuntu@smithi044.front.sepia.ceph.com
2021-03-02T04:29:39.302 INFO:scrape:ceph version 17.0.0-1318-ge994ad91 (e994ad91b4985b37c54012a8b608a99fd61198fc) quincy (dev)
 1: /lib64/libpthread.so.0(+0x12b20) [0x7fb78bea0b20]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x55895f2e3b83]
 5: ceph-osd(+0x567d4c) [0x55895f2e3d4c]
 6: (SharedLRU<unsigned int, OSDMap const>::~SharedLRU()+0x353) [0x55895f471dc3]
 7: (OSDService::~OSDService()+0xd9) [0x55895f476f19]
 8: (OSD::~OSD()+0x21c) [0x55895f421f4c]
 9: (OSD::~OSD()+0xd) [0x55895f4221fd]
 10: main()
 11: __libc_start_main()
 12: _start()
2021-03-02T04:29:39.302 INFO:scrape:1 jobs: ['5926062']
2021-03-02T04:29:39.303 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs:verify/{begin', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag_enable', 'ranks/5', 'session_timeout', 'tasks/fsstress', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-03-02T04:29:39.304 INFO:scrape:
2021-03-02T04:29:39.304 INFO:scrape:Failure: "2021-03-02T03:00:34.591789+0000 mds.b (mds.0) 2 : cluster [WRN] slow request 184.925722 seconds old, received at 2021-03-02T02:57:29.648031+0000: client_request(client.4567:17482 rmdir #0x1000000322a/d3a 2021-03-02T02:57:29.646995+0000 caller_uid=1000, caller_gid=1236{6,36,1000,1236,}) currently failed to rdlock, waiting" in cluster log
2021-03-02T04:29:39.304 INFO:scrape:1 jobs: ['5926060']
2021-03-02T04:29:39.305 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs:verify/{begin', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'ranks/1', 'session_timeout', 'tasks/fsstress', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-03-02T04:29:39.305 INFO:scrape:
2021-03-02T04:29:39.306 INFO:scrape:Failure: Command failed on smithi014 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'
2021-03-02T04:29:39.306 INFO:scrape:1 jobs: ['5926064']
2021-03-02T04:29:39.307 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:verify/{begin', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'ranks/3', 'session_timeout', 'tasks/fsstress', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-03-02T04:29:39.307 INFO:scrape:
2021-03-02T04:29:39.308 INFO:scrape:Failure: "2021-03-02T03:00:53.475664+0000 mds.d (mds.0) 2 : cluster [WRN] slow request 182.214916 seconds old, received at 2021-03-02T02:57:51.238959+0000: internal op exportdir:mds.0:275 currently failed to wrlock, waiting" in cluster log
2021-03-02T04:29:39.308 INFO:scrape:1 jobs: ['5926058']
2021-03-02T04:29:39.309 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs:verify/{begin', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag_enable', 'ranks/3', 'session_timeout', 'tasks/fsstress', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-03-02T04:29:39.309 INFO:scrape:
2021-03-02T04:29:39.311 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2021-03-02_01:44:39-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/pdonnell-2021-03-02_01:44:39-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-02_01:44:39-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/
failed:  4
dead:    0
running: 0
waiting: 0
queued:  0
passed:  6


Fail
=================================================================
[5926058]  fs:verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   02:13:11
info:   http://pulpito.ceph.com/pdonnell-2021-03-02_01:44:39-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5926058/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-02_01:44:39-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5926058/

    "2021-03-02T03:00:53.475664+0000 mds.d (mds.0) 2 : cluster [WRN] slow
    request 182.214916 seconds old, received at
    2021-03-02T02:57:51.238959+0000: internal op exportdir:mds.0:275 currently
    failed to wrlock, waiting" in cluster log

[5926060]  fs:verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   01:44:49
info:   http://pulpito.ceph.com/pdonnell-2021-03-02_01:44:39-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5926060/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-02_01:44:39-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5926060/

    "2021-03-02T03:00:34.591789+0000 mds.b (mds.0) 2 : cluster [WRN] slow
    request 184.925722 seconds old, received at
    2021-03-02T02:57:29.648031+0000: client_request(client.4567:17482 rmdir
    #0x1000000322a/d3a 2021-03-02T02:57:29.646995+0000 caller_uid=1000,
    caller_gid=1236{6,36,1000,1236,}) currently failed to rdlock, waiting" in
    cluster log

[5926062]  fs:verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   02:29:53
info:   http://pulpito.ceph.com/pdonnell-2021-03-02_01:44:39-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5926062/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-02_01:44:39-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5926062/

    Found coredumps on ubuntu@smithi044.front.sepia.ceph.com

[5926064]  fs:verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   01:54:04
info:   http://pulpito.ceph.com/pdonnell-2021-03-02_01:44:39-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5926064/
log:    http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-02_01:44:39-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5926064/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d974df2878364d279b7b6862809de137

    Command failed on smithi014 with status 1: 'sudo ceph --cluster ceph osd
    crush tunables default'



Pass
=================================================================
[5926057] fs:verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/dbench validater/lockdep}
time:   00:46:19
info:   http://pulpito.ceph.com/pdonnell-2021-03-02_01:44:39-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5926057/

[5926059] fs:verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/dbench validater/lockdep}
time:   01:31:24
info:   http://pulpito.ceph.com/pdonnell-2021-03-02_01:44:39-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5926059/

[5926061] fs:verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/dbench validater/lockdep}
time:   01:11:33
info:   http://pulpito.ceph.com/pdonnell-2021-03-02_01:44:39-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5926061/

[5926063] fs:verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/dbench validater/lockdep}
time:   00:55:21
info:   http://pulpito.ceph.com/pdonnell-2021-03-02_01:44:39-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5926063/

[5926065] fs:verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/dbench validater/lockdep}
time:   01:04:12
info:   http://pulpito.ceph.com/pdonnell-2021-03-02_01:44:39-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5926065/

[5926066] fs:verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/fsstress validater/valgrind}
time:   01:40:46
info:   http://pulpito.ceph.com/pdonnell-2021-03-02_01:44:39-fs:verify-wip-pdonnell-testing-20210301.234318-distro-basic-smithi/5926066/