2023-09-26T19:23:41.031 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-26T19:23:41.032 INFO:teuthology.results:subset: '111/120000'
2023-09-26T19:23:41.032 INFO:teuthology.results:seed: '2476'
2023-09-26T20:54:18.801 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-26T20:54:18.831 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-09-26T21:48:22.731 INFO:teuthology.results:Tests finished! gathering results...
2023-09-26T21:48:24.463 INFO:scrape:Found 30 jobs
2023-09-26T21:48:24.476 INFO:scrape:Found 1 distinct failure reasons
2023-09-26T21:48:24.476 INFO:scrape:Failure: timeout expired in wait_for_all_osds_up
2023-09-26T21:48:24.476 INFO:scrape:30 jobs
2023-09-26T21:48:24.476 INFO:scrape:suites intersection: ['clusters/{fixed-2', 'fast/fast', 'mon_election/classic', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados:thrash-erasure-code/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/minsize_recovery', 'thrashosds-health', 'workloads/ec-rados-plugin=clay-k=4-m=2}']
2023-09-26T21:48:24.477 INFO:scrape:suites union: ['clusters/{fixed-2', 'fast/fast', 'mon_election/classic', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados:thrash-erasure-code/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/minsize_recovery', 'thrashosds-health', 'workloads/ec-rados-plugin=clay-k=4-m=2}']
2023-09-26T21:48:24.477 INFO:scrape:
2023-09-26T21:48:24.477 INFO:teuthology.results:Sending results to ksirivad@redhat.com: Test Run: ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/
failed:  30
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7402551]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:46:15
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402551/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402551/

    timeout expired in wait_for_all_osds_up

[7402552]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:44:59
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402552/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402552/

    timeout expired in wait_for_all_osds_up

[7402553]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:45:42
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402553/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402553/

    timeout expired in wait_for_all_osds_up

[7402554]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:44:46
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402554/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402554/

    timeout expired in wait_for_all_osds_up

[7402555]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:45:18
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402555/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402555/

    timeout expired in wait_for_all_osds_up

[7402556]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:46:31
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402556/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402556/

    timeout expired in wait_for_all_osds_up

[7402557]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:45:01
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402557/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402557/

    timeout expired in wait_for_all_osds_up

[7402558]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:45:31
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402558/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402558/

    timeout expired in wait_for_all_osds_up

[7402559]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:45:47
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402559/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402559/

    timeout expired in wait_for_all_osds_up

[7402560]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:45:24
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402560/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402560/

    timeout expired in wait_for_all_osds_up

[7402561]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:45:04
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402561/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402561/

    timeout expired in wait_for_all_osds_up

[7402562]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:45:59
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402562/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402562/

    timeout expired in wait_for_all_osds_up

[7402563]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:45:33
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402563/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402563/

    timeout expired in wait_for_all_osds_up

[7402564]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:47:37
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402564/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402564/

    timeout expired in wait_for_all_osds_up

[7402565]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:45:03
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402565/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402565/

    timeout expired in wait_for_all_osds_up

[7402566]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:47:01
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402566/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402566/

    timeout expired in wait_for_all_osds_up

[7402567]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:46:06
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402567/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402567/

    timeout expired in wait_for_all_osds_up

[7402568]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:51:14
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402568/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402568/

    timeout expired in wait_for_all_osds_up

[7402569]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:45:30
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402569/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402569/

    timeout expired in wait_for_all_osds_up

[7402570]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:46:38
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402570/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402570/

    timeout expired in wait_for_all_osds_up

[7402571]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:45:52
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402571/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402571/

    timeout expired in wait_for_all_osds_up

[7402572]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:45:35
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402572/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402572/

    timeout expired in wait_for_all_osds_up

[7402573]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:46:32
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402573/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402573/

    timeout expired in wait_for_all_osds_up

[7402574]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:45:10
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402574/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402574/

    timeout expired in wait_for_all_osds_up

[7402575]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:45:39
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402575/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402575/

    timeout expired in wait_for_all_osds_up

[7402576]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:47:02
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402576/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402576/

    timeout expired in wait_for_all_osds_up

[7402577]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:49:31
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402577/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402577/

    timeout expired in wait_for_all_osds_up

[7402578]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:49:32
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402578/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402578/

    timeout expired in wait_for_all_osds_up

[7402579]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:44:59
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402579/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402579/

    timeout expired in wait_for_all_osds_up

[7402580]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
-----------------------------------------------------------------
time:   00:45:34
info:   http://pulpito.ceph.com/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402580/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-26_19:22:00-rados:thrash-erasure-code-main-distro-default-smithi/7402580/

    timeout expired in wait_for_all_osds_up