2023-09-29T18:51:58.819 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-29T18:51:58.819 INFO:teuthology.results:seed: '1929'
2023-09-29T18:52:42.846 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-29T18:52:42.852 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-09-29T20:23:49.645 INFO:teuthology.results:Tests finished! gathering results...
2023-09-29T20:23:49.799 INFO:scrape:Found 1 jobs
2023-09-29T20:23:50.242 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.243 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.243 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.243 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.261 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.261 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.261 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.262 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.552 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.552 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.552 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.552 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.558 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.558 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.558 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.558 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.810 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.810 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.811 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.811 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.824 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.824 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.825 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.825 WARNING:scrape:Saw end of BT but not start
2023-09-29T20:23:50.886 INFO:scrape:Found 1 distinct failure reasons
2023-09-29T20:23:50.886 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'}
2023-09-29T20:23:50.886 INFO:scrape:1 jobs: ['7406456']
2023-09-29T20:23:50.887 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}']
2023-09-29T20:23:50.887 INFO:scrape:
2023-09-29T20:23:50.887 INFO:teuthology.results:Sending results to choffman@redhat.com: Test Run: choffman-2023-09-29_18:50:31-fs:workload-wip-choffman-62793-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/choffman-2023-09-29_18:50:31-fs:workload-wip-choffman-62793-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/choffman-2023-09-29_18:50:31-fs:workload-wip-choffman-62793-distro-default-smithi/
failed:  1
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7406456]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}}
-----------------------------------------------------------------
time:   01:25:11
info:   http://pulpito.ceph.com/choffman-2023-09-29_18:50:31-fs:workload-wip-choffman-62793-distro-default-smithi/7406456/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-09-29_18:50:31-fs:workload-wip-choffman-62793-distro-default-smithi/7406456/

    error during scrub thrashing: rank damage found: {'backtrace'}