2020-12-02T23:35:28.920 INFO:root:teuthology version: 1.0.0-e6b33b7b
2020-12-02T23:35:28.922 INFO:teuthology.results:seed: '7009'
2020-12-03T00:25:34.474 INFO:root:teuthology version: 1.0.0-e6b33b7b
2020-12-03T00:25:34.591 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-12-03T12:12:46.916 INFO:teuthology.results:Tests finished! gathering results...
2020-12-03T12:12:48.817 INFO:scrape:Found 20 jobs
2020-12-03T12:12:53.098 INFO:scrape:Found 3 distinct failure reasons
2020-12-03T12:12:53.099 INFO:scrape:Failure: '/home/ubuntu/cephtest/archive/syslog/kern.log:2020-12-03T00:01:57.604604+00:00 gibba014 kernel: BUG: sleeping function called from invalid context at mm/slab.h:506
' in syslog
2020-12-03T12:12:53.099 INFO:scrape:6 jobs: ['5675924', '5675930', '5675922', '5675928', '5675925', '5675934']
2020-12-03T12:12:53.100 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2020-12-03T12:12:53.100 INFO:scrape:suites union: ['clusters/1a2s-mds-1c-client-3node', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/snaps/{begin', 'fs/workload/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore-ec/filestore-xfs', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'session_timeout', 'tasks/workunit/fs/misc}', 'tasks/workunit/snaps}', 'tasks/workunit/suites/blogbench}', 'tasks/workunit/suites/fsstress}', 'tasks/workunit/suites/pjd}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2020-12-03T12:12:53.101 INFO:scrape:
2020-12-03T12:12:53.101 INFO:scrape:Dead: b'2020-12-03T11:35:39.577 DEBUG:teuthology.exit:Finished running handlers'
2020-12-03T12:12:53.102 INFO:scrape:6 jobs: ['5675918', '5675935', '5675921', '5675926', '5675931', '5675929']
2020-12-03T12:12:53.102 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2020-12-03T12:12:53.103 INFO:scrape:suites union: ['clusters/1-mds-4-client-coloc', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'fs/volumes/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'tasks/admin}', 'tasks/backtrace}', 'tasks/client-recovery}', 'tasks/fragment}', 'tasks/mds_creation_retry}', 'tasks/volumes/{overrides', 'test/basic}}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2020-12-03T12:12:53.103 INFO:scrape:
2020-12-03T12:12:53.104 INFO:scrape:Failure: 
2020-12-03T12:12:53.104 INFO:scrape:1 jobs: ['5675917']
2020-12-03T12:12:53.104 INFO:scrape:
2020-12-03T12:12:53.106 INFO:teuthology.results:Sending results to jlayton@redhat.com: Test Run: jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba
=================================================================
info:    http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/
logs:    http://qa-proxy.ceph.com/teuthology/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/
failed:  6
dead:    7
running: 0
waiting: 0
queued:  0
passed:  7


Fail
=================================================================
[5675922]  fs/workload/{begin clusters/1a2s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} tasks/workunit/suites/pjd}
-----------------------------------------------------------------
time:   00:10:48
info:   http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675922/
log:    http://qa-proxy.ceph.com/teuthology/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675922/

    '/home/ubuntu/cephtest/archive/syslog/kern.log:2020-12-03T00:02:58.120820+0
    0:00 gibba021 kernel: BUG: sleeping function called from invalid context at
    mm/slab.h:506 ' in syslog

[5675924]  fs/workload/{begin clusters/1a2s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} tasks/workunit/suites/blogbench}
-----------------------------------------------------------------
time:   00:19:23
info:   http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675924/
log:    http://qa-proxy.ceph.com/teuthology/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675924/

    '/home/ubuntu/cephtest/archive/syslog/kern.log:2020-12-03T00:01:57.604604+0
    0:00 gibba014 kernel: BUG: sleeping function called from invalid context at
    mm/slab.h:506 ' in syslog

[5675925]  fs/workload/{begin clusters/1a2s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} tasks/workunit/suites/fsstress}
-----------------------------------------------------------------
time:   00:12:18
info:   http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675925/
log:    http://qa-proxy.ceph.com/teuthology/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675925/

    '/home/ubuntu/cephtest/archive/syslog/kern.log:2020-12-03T00:13:36.767217+0
    0:00 gibba045 kernel: BUG: sleeping function called from invalid context at
    mm/slab.h:506 ' in syslog

[5675928]  fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:49:19
info:   http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675928/
log:    http://qa-proxy.ceph.com/teuthology/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675928/

    '/home/ubuntu/cephtest/archive/syslog/kern.log:2020-12-03T00:00:40.034271+0
    0:00 gibba003 kernel: BUG: sleeping function called from invalid context at
    mm/slab.h:506 ' in syslog

[5675930]  fs/workload/{begin clusters/1a2s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/filestore-xfs omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} tasks/workunit/fs/misc}
-----------------------------------------------------------------
time:   00:23:50
info:   http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675930/
log:    http://qa-proxy.ceph.com/teuthology/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675930/

    '/home/ubuntu/cephtest/archive/syslog/kern.log:2020-12-03T00:19:06.545500+0
    0:00 gibba020 kernel: BUG: sleeping function called from invalid context at
    mm/slab.h:506 ' in syslog

[5675934]  fs/workload/{begin clusters/1a2s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} tasks/workunit/suites/pjd}
-----------------------------------------------------------------
time:   00:10:26
info:   http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675934/
log:    http://qa-proxy.ceph.com/teuthology/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675934/

    '/home/ubuntu/cephtest/archive/syslog/kern.log:2020-12-03T00:35:34.901178+0
    0:00 gibba008 kernel: BUG: sleeping function called from invalid context at
    mm/slab.h:506 ' in syslog



Dead
=================================================================
[5675917]  fs/workload/{begin clusters/1a2s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} tasks/workunit/fs/misc}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675917/
log:    http://qa-proxy.ceph.com/teuthology/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675917/
sentry: https://sentry.ceph.com/organizations/ceph/?query=210cdd0ed2a44bc6b4068005bbc124c2

[5675918]  fs/functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675918/
log:    http://qa-proxy.ceph.com/teuthology/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675918/

[5675921]  fs/functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/backtrace}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675921/
log:    http://qa-proxy.ceph.com/teuthology/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675921/

[5675926]  fs/functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-recovery}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675926/
log:    http://qa-proxy.ceph.com/teuthology/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675926/

[5675929]  fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675929/
log:    http://qa-proxy.ceph.com/teuthology/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675929/

[5675931]  fs/functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/fragment}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675931/
log:    http://qa-proxy.ceph.com/teuthology/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675931/

[5675935]  fs/functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds_creation_retry}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675935/
log:    http://qa-proxy.ceph.com/teuthology/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675935/



Pass
=================================================================
[5675916] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{frag_enable osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:22:52
info:   http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675916/

[5675919] fs/workload/{begin clusters/1a2s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} tasks/workunit/suites/ffsb}
time:   00:31:13
info:   http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675919/

[5675920] fs/workload/{begin clusters/1a2s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/filestore-xfs omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} tasks/workunit/suites/fsync-tester}
time:   00:09:20
info:   http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675920/

[5675923] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth}
time:   00:17:31
info:   http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675923/

[5675927] fs/workload/{begin clusters/1a2s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} tasks/workunit/suites/iogen}
time:   00:41:14
info:   http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675927/

[5675932] fs/workload/{begin clusters/1a2s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} tasks/workunit/suites/ffsb}
time:   00:58:20
info:   http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675932/

[5675933] fs/workload/{begin clusters/1a2s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} tasks/workunit/suites/fsync-tester}
time:   00:09:15
info:   http://pulpito.ceph.com/jlayton-2020-12-02_23:35:05-fs-master-wip-fscache-iter-basic-gibba/5675933/