2023-10-02T15:52:47.102 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-10-02T15:52:47.103 INFO:teuthology.results:seed: '5406'
2023-10-02T15:53:27.703 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-10-02T15:53:27.706 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-10-02T17:39:34.143 INFO:teuthology.results:Tests finished! gathering results...
2023-10-02T17:39:34.365 INFO:scrape:Found 1 jobs
2023-10-02T17:39:34.767 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:34.768 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:34.768 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:34.768 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:34.776 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:34.777 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:34.778 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:34.778 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:34.786 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:34.789 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:34.790 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:34.790 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:34.797 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:34.797 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:34.797 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:34.797 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.140 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.141 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.141 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.141 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.152 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.152 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.152 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.152 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.158 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.158 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.159 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.159 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.166 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.166 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.166 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.166 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.442 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.442 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.443 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.443 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.450 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.450 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.450 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.451 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.455 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.456 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.456 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.456 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.461 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.462 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.462 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.462 WARNING:scrape:Saw end of BT but not start
2023-10-02T17:39:35.524 INFO:scrape:Found 1 distinct failure reasons
2023-10-02T17:39:35.524 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'}
2023-10-02T17:39:35.524 INFO:scrape:1 jobs: ['7408507']
2023-10-02T17:39:35.525 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs:workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}']
2023-10-02T17:39:35.525 INFO:scrape:
2023-10-02T17:39:35.525 INFO:teuthology.results:Sending results to choffman@redhat.com: Test Run: choffman-2023-10-02_15:52:23-fs:workload-wip-choffman-62793-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/choffman-2023-10-02_15:52:23-fs:workload-wip-choffman-62793-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/choffman-2023-10-02_15:52:23-fs:workload-wip-choffman-62793-distro-default-smithi/
failed:  1
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7408507]  fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}}
-----------------------------------------------------------------
time:   01:36:39
info:   http://pulpito.ceph.com/choffman-2023-10-02_15:52:23-fs:workload-wip-choffman-62793-distro-default-smithi/7408507/
log:    http://qa-proxy.ceph.com/teuthology/choffman-2023-10-02_15:52:23-fs:workload-wip-choffman-62793-distro-default-smithi/7408507/

    error during scrub thrashing: rank damage found: {'backtrace'}