2023-09-28T21:26:30.726 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-28T21:26:30.726 INFO:teuthology.results:subset: '111/120000'
2023-09-28T21:26:30.726 INFO:teuthology.results:seed: '9918'
2023-09-28T22:08:50.802 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-28T22:08:50.825 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-09-29T00:33:02.168 INFO:teuthology.results:Tests finished! gathering results...
2023-09-29T00:33:04.226 INFO:scrape:Found 27 jobs
2023-09-29T00:33:06.599 INFO:scrape:Found 2 distinct failure reasons
2023-09-29T00:33:06.599 INFO:scrape:Timeout 120 running ceph --cluster ceph osd dump --format=json
2023-09-29T00:33:06.599 INFO:scrape:1 jobs: ['7405734']
2023-09-29T00:33:06.599 INFO:scrape:suites: ['clusters/{fixed-2', 'fast/fast', 'mon_election/classic', 'msgr-failures/fastclose', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados:thrash-erasure-code/{ceph', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{rhel_8}', 'thrashers/minsize_recovery', 'thrashosds-health', 'workloads/ec-small-objects-many-deletes}']
2023-09-29T00:33:06.600 INFO:scrape:
2023-09-29T00:33:06.600 INFO:scrape:Failure: Command failed on smithi094 with status 123: "time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --"
2023-09-29T00:33:06.600 INFO:scrape:1 jobs: ['7405738']
2023-09-29T00:33:06.600 INFO:scrape:suites: ['clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados:thrash-erasure-code/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-small-objects}']
2023-09-29T00:33:06.600 INFO:scrape:
2023-09-29T00:33:06.601 INFO:teuthology.results:Sending results to ksirivad@redhat.com: Test Run: ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/
failed:  2
dead:    0
running: 0
waiting: 0
queued:  0
passed:  25


Fail
=================================================================
[7405734]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-many-deletes}
-----------------------------------------------------------------
time:   00:41:51
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405734/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405734/

    Command failed on smithi133 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph osd dump --format=json'

[7405738]  rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects}
-----------------------------------------------------------------
time:   00:20:04
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405738/
log:    http://qa-proxy.ceph.com/teuthology/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405738/

    Command failed on smithi094 with status 123: "time sudo find /var/log/ceph
    -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0
    --no-run-if-empty -- gzip -5 --verbose --"



Pass
=================================================================
[7405712] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects}
time:   00:20:27
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405712/

[7405713] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects}
time:   00:21:35
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405713/

[7405714] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects}
time:   00:20:44
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405714/

[7405715] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
time:   00:45:02
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405715/

[7405716] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
time:   00:44:55
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405716/

[7405717] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
time:   00:45:40
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405717/

[7405718] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1}
time:   00:24:52
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405718/

[7405719] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1}
time:   00:24:04
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405719/

[7405720] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1}
time:   00:27:11
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405720/

[7405721] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1}
time:   00:25:40
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405721/

[7405722] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1}
time:   00:28:33
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405722/

[7405723] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1}
time:   00:24:41
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405723/

[7405724] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-radosbench}
time:   01:19:46
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405724/

[7405725] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-radosbench}
time:   02:27:29
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405725/

[7405726] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-radosbench}
time:   01:29:37
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405726/

[7405727] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-small-objects-balanced}
time:   00:23:52
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405727/

[7405728] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-small-objects-balanced}
time:   00:25:02
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405728/

[7405729] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-small-objects-balanced}
time:   00:25:39
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405729/

[7405730] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/fastread thrashosds-health workloads/ec-small-objects-fast-read}
time:   00:20:57
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405730/

[7405731] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/fastread thrashosds-health workloads/ec-small-objects-fast-read}
time:   00:21:38
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405731/

[7405732] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/fastread thrashosds-health workloads/ec-small-objects-fast-read}
time:   00:20:54
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405732/

[7405733] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-many-deletes}
time:   00:49:21
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405733/

[7405735] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-many-deletes}
time:   00:50:23
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405735/

[7405736] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects}
time:   00:20:13
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405736/

[7405737] rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects}
time:   00:20:21
info:   http://pulpito.ceph.com/ksirivad-2023-09-28_20:55:56-rados:thrash-erasure-code-main-distro-default-smithi/7405737/