2020-12-07T15:47:09.977 INFO:root:teuthology version: 1.0.0-e6b33b7b 2020-12-07T15:47:09.978 INFO:teuthology.results:seed: '154' 2020-12-07T16:21:36.727 INFO:root:teuthology version: 1.0.0-e6b33b7b 2020-12-07T16:21:36.789 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-12-07T17:30:37.990 INFO:teuthology.results:Tests finished! gathering results... 2020-12-07T17:30:39.933 INFO:scrape:Found 20 jobs 2020-12-07T17:30:40.262 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689758/teuthology.log matches 2020-12-07T17:30:40.263 WARNING:scrape:Not-understood crash indication 2020-12-07T17:30:40.593 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689758/teuthology.log matches 2020-12-07T17:30:40.594 WARNING:scrape:Not-understood crash indication 2020-12-07T17:30:40.595 INFO:scrape:Found 1 distinct failure reasons 2020-12-07T17:30:40.596 INFO:scrape:Timeout 120 running ceph --cluster ceph tell osd.7 flush_pg_stats 2020-12-07T17:30:40.596 INFO:scrape:1 jobs: ['5689758'] 2020-12-07T17:30:40.597 INFO:scrape:suites: ['2-workunit/cfuse_workunit_suites_fsstress}}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2020-12-07T17:30:40.597 INFO:scrape: 2020-12-07T17:30:40.599 INFO:teuthology.results:Sending results to jlayton@redhat.com: Test Run: jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba ================================================================= info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/ logs: http://qa-proxy.ceph.com/teuthology/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/ failed: 1 dead: 0 running: 0 waiting: 0 queued: 0 passed: 19 Fail ================================================================= [5689758] fs/thrash/workloads/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/osd 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:18:44 info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689758/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689758/ Command failed on gibba018 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats' Pass ================================================================= [5689749] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{frag_enable osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} time: 01:25:28 info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689749/ [5689750] fs/workload/{begin clusters/1a2s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} tasks/workunit/fs/misc} time: 00:27:22 info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689750/ [5689751] fs/thrash/workloads/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:15:24 info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689751/ [5689752] fs/functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin} time: 00:21:50 info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689752/ [5689753] fs/thrash/workloads/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/osd 2-workunit/iozone}} time: 00:12:36 info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689753/ [5689754] fs/workload/{begin clusters/1a2s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} tasks/workunit/suites/ffsb} time: 00:33:57 info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689754/ [5689755] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:17:38 info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689755/ [5689756] fs/thrash/workloads/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/osd 2-workunit/cfuse_workunit_trivial_sync}} time: 00:08:17 info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689756/ [5689757] fs/workload/{begin clusters/1a2s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/filestore-xfs omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} tasks/workunit/suites/fsync-tester} time: 00:09:31 info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689757/ [5689759] fs/functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/backtrace} time: 00:07:56 info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689759/ [5689760] fs/workload/{begin clusters/1a2s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} tasks/workunit/suites/pjd} time: 00:10:35 info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689760/ [5689761] fs/thrash/workloads/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:14:41 info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689761/ [5689762] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} time: 00:16:48 info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689762/ [5689763] fs/thrash/workloads/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:07:56 info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689763/ [5689764] fs/workload/{begin clusters/1a2s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} tasks/workunit/suites/blogbench} time: 00:23:21 info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689764/ [5689765] fs/thrash/workloads/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:15:33 info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689765/ [5689766] fs/workload/{begin clusters/1a2s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} tasks/workunit/suites/fsstress} time: 00:10:39 info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689766/ [5689767] fs/functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-recovery} time: 00:26:00 info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689767/ [5689768] fs/thrash/workloads/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:19:35 info: http://pulpito.ceph.com/jlayton-2020-12-07_15:46:26-fs-master-wip-fscache-iter-basic-gibba/5689768/