2023-03-22T21:07:28.546 INFO:root:teuthology version: 0.0.1.dev66+g73b89aa
2023-03-22T21:07:28.547 INFO:teuthology.results:seed: '4733'
2023-03-22T21:09:06.690 INFO:root:teuthology version: 0.0.1.dev66+g73b89aa
2023-03-22T21:09:06.694 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-03-23T09:10:03.194 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2023-03-23T09:10:03.532 INFO:scrape:Found 3 jobs
2023-03-23T09:10:03.609 INFO:scrape:Found 3 distinct failure reasons
2023-03-23T09:10:03.610 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/ffsb.sh
2023-03-23T09:10:03.610 INFO:scrape:1 jobs: ['7216331']
2023-03-23T09:10:03.610 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'wsync/no}']
2023-03-23T09:10:03.611 INFO:scrape:
2023-03-23T09:10:03.611 INFO:scrape:Dead: b'2023-03-23T09:10:03.566 INFO:journalctl@ceph.mon.c.smithi131.stdout:Mar 23 09:10:03 smithi131 ceph-mon[94726]: Health check update: 1024 slow ops, oldest one blocked for 40872 sec, daemons [osd.0,osd.10,osd.6,osd.8] have slow ops. (SLOW_OPS)'
2023-03-23T09:10:03.611 INFO:scrape:1 jobs: ['7216332']
2023-03-23T09:10:03.612 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}']
2023-03-23T09:10:03.612 INFO:scrape:
2023-03-23T09:10:03.612 INFO:scrape:Dead: b'2023-03-23T09:10:03.342 INFO:journalctl@ceph.osd.9.smithi044.stdout:Mar 23 09:10:02 smithi044 ceph-e0dcb46a-c8f7-11ed-9afe-001a4aab830c-osd-9[122230]: 2023-03-23T09:10:02.925+0000 7f2562042700 -1 osd.9 371 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.25082.0:3833 3.3d 3.b9df5cbd (undecoded) ondisk+retry+write+known_if_redirected e187)'
2023-03-23T09:10:03.613 INFO:scrape:1 jobs: ['7216330']
2023-03-23T09:10:03.613 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}']
2023-03-23T09:10:03.613 INFO:scrape: