2026-02-23T19:04:18.188 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-23T19:04:18.188 INFO:teuthology.results:subset: '1/500' 2026-02-23T19:04:18.188 INFO:teuthology.results:seed: '3302' 2026-02-23T19:09:58.598 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-23T19:09:58.605 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2026-02-23T19:56:05.277 INFO:teuthology.results:Tests finished! gathering results... 2026-02-23T19:56:05.644 INFO:scrape:Found 15 jobs 2026-02-23T19:56:05.644 INFO:scrape:Found 1 distinct failure reasons 2026-02-23T19:56:05.644 INFO:scrape:Failure: Command failed on trial198 with status 1: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph_test_rados --no-omap --ec-pool --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' 2026-02-23T19:56:05.644 INFO:scrape:1 jobs: ['66847'] 2026-02-23T19:56:05.644 INFO:scrape:suites: ['base', 'clusters/{fixed-4}', 'ec_direct_reads/ec_direct_reads_on', 'ec_optimizations/ec_optimizations_on', 'fast/normal', 'mem$/{low}', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'objectstore/{bluestore/{alloc$/{avl}', 'onode-segment$/{512K}', 'rados', 'rados/thrash-erasure-code/{ceph', 'random}}}}', 'recovery-overrides/{default}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=8-m=6-crush}', 'write$/{random/{compr$/{yes$/{zstd}}'] 2026-02-23T19:56:05.644 INFO:scrape: 2026-02-23T19:56:05.645 INFO:teuthology.results:Sending results to callum.james@ibm.com: Test Run: cjames-2026-02-23_16:19:20-rados-main-distro-default-trial ================================================================= info: https://pulpito.ceph.com/cjames-2026-02-23_16:19:20-rados-main-distro-default-trial/ logs: https://qa-proxy.ceph.com/teuthology/cjames-2026-02-23_16:19:20-rados-main-distro-default-trial/ failed: 1 dead: 0 running: 0 waiting: 0 queued: 0 passed: 14 Fail ================================================================= [66847] rados/thrash-erasure-code/{ceph clusters/{fixed-4} ec_direct_reads/ec_direct_reads_on ec_optimizations/ec_optimizations_on fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore/{alloc$/{avl} base mem$/{low} onode-segment$/{512K} write$/{random/{compr$/{yes$/{zstd}} random}}}} rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} ----------------------------------------------------------------- time: 00:21:05 info: https://pulpito.ceph.com/cjames-2026-02-23_16:19:20-rados-main-distro-default-trial/66847/ log: https://qa-proxy.ceph.com/teuthology/cjames-2026-02-23_16:19:20-rados-main-distro-default-trial/66847/ Command failed on trial198 with status 1: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph_test_rados --no-omap --ec-pool --max-ops 400000 --objects 1024 --max- in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' Pass ================================================================= [66839] rados/thrash-erasure-code/{ceph clusters/{fixed-4} ec_direct_reads/ec_direct_reads_on ec_optimizations/ec_optimizations_on fast/normal mon_election/connectivity msgr-failures/few objectstore/{bluestore/{alloc$/{bitmap} base mem$/{normal-1} onode-segment$/{256K} write$/{random/{compr$/{yes$/{zlib}} random}}}} rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read} time: 00:17:35 info: https://pulpito.ceph.com/cjames-2026-02-23_16:19:20-rados-main-distro-default-trial/66839/ [66840] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph clusters/{fixed-4} ec_direct_reads/ec_direct_reads_on ec_optimizations/ec_optimizations_on mon_election/connectivity msgr-failures/few objectstore/{bluestore/{alloc$/{stupid} base mem$/{normal-2} onode-segment$/{512K-onoff} write$/{v1/{compr$/{no$/{no}} v1}}}} rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} time: 00:18:08 info: https://pulpito.ceph.com/cjames-2026-02-23_16:19:20-rados-main-distro-default-trial/66840/ [66841] rados/thrash-erasure-code-big/{ceph cluster/{12-osds} ec_direct_reads/ec_direct_reads_on ec_optimizations/ec_optimizations_on mon_election/connectivity msgr-failures/few objectstore/{bluestore/{alloc$/{btree} base mem$/{normal-2} onode-segment$/{512K} write$/{v2/{compr$/{no$/{no}} v2}}}} rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:08:22 info: https://pulpito.ceph.com/cjames-2026-02-23_16:19:20-rados-main-distro-default-trial/66841/ [66842] rados/thrash-erasure-code/{ceph clusters/{fixed-4} ec_direct_reads/ec_direct_reads_on ec_optimizations/ec_optimizations_on fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore/{alloc$/{stupid} base mem$/{low} onode-segment$/{none} write$/{random/{compr$/{yes$/{snappy}} random}}}} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects} time: 00:38:42 info: https://pulpito.ceph.com/cjames-2026-02-23_16:19:20-rados-main-distro-default-trial/66842/ [66843] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph clusters/{fixed-4} ec_direct_reads/ec_direct_reads_on ec_optimizations/ec_optimizations_on mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore/{alloc$/{stupid} base mem$/{normal-2} onode-segment$/{256K} write$/{random/{compr$/{yes$/{zlib}} random}}}} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap_host thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} time: 00:23:59 info: https://pulpito.ceph.com/cjames-2026-02-23_16:19:20-rados-main-distro-default-trial/66843/ [66844] rados/thrash-erasure-code/{ceph clusters/{fixed-4} ec_direct_reads/ec_direct_reads_on ec_optimizations/ec_optimizations_on fast/normal mon_election/connectivity msgr-failures/few objectstore/{bluestore/{alloc$/{bitmap} base mem$/{normal-2} onode-segment$/{256K} write$/{v1/{compr$/{yes$/{lz4}} v1}}}} rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:21:27 info: https://pulpito.ceph.com/cjames-2026-02-23_16:19:20-rados-main-distro-default-trial/66844/ [66845] rados/thrash-erasure-code-big/{ceph cluster/{12-osds} ec_direct_reads/ec_direct_reads_on ec_optimizations/ec_optimizations_on mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore/{alloc$/{bitmap} base mem$/{normal-2} onode-segment$/{none} write$/{v2/{compr$/{no$/{no}} v2}}}} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2-cs=16k} time: 00:25:59 info: https://pulpito.ceph.com/cjames-2026-02-23_16:19:20-rados-main-distro-default-trial/66845/ [66846] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph clusters/{fixed-4} ec_direct_reads/ec_direct_reads_on ec_optimizations/ec_optimizations_on mon_election/connectivity msgr-failures/few objectstore/{bluestore/{alloc$/{hybrid} base mem$/{normal-1} onode-segment$/{1M} write$/{random/{compr$/{yes$/{lz4}} random}}}} rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} time: 00:20:16 info: https://pulpito.ceph.com/cjames-2026-02-23_16:19:20-rados-main-distro-default-trial/66846/ [66848] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph clusters/{fixed-4} ec_direct_reads/ec_direct_reads_on ec_optimizations/ec_optimizations_on mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore/{alloc$/{bitmap} base mem$/{low} onode-segment$/{none} write$/{random/{compr$/{no$/{no}} random}}}} rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow_host thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} time: 00:21:09 info: https://pulpito.ceph.com/cjames-2026-02-23_16:19:20-rados-main-distro-default-trial/66848/ [66849] rados/thrash-erasure-code-big/{ceph cluster/{12-osds} ec_direct_reads/ec_direct_reads_on ec_optimizations/ec_optimizations_on mon_election/connectivity msgr-failures/few objectstore/{bluestore/{alloc$/{btree} base mem$/{normal-1} onode-segment$/{512K} write$/{v2/{compr$/{yes$/{lz4}} v2}}}} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:22:36 info: https://pulpito.ceph.com/cjames-2026-02-23_16:19:20-rados-main-distro-default-trial/66849/ [66850] rados/thrash-erasure-code/{ceph clusters/{fixed-4} ec_direct_reads/ec_direct_reads_on ec_optimizations/ec_optimizations_on fast/normal mon_election/connectivity msgr-failures/few objectstore/{bluestore/{alloc$/{hybrid} base mem$/{normal-1} onode-segment$/{512K} write$/{v1/{compr$/{yes$/{zlib}} v1}}}} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-fast-read} time: 00:39:32 info: https://pulpito.ceph.com/cjames-2026-02-23_16:19:20-rados-main-distro-default-trial/66850/ [66851] rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph clusters/{fixed-4} ec_direct_reads/ec_direct_reads_on ec_optimizations/ec_optimizations_on mon_election/connectivity msgr-failures/fastclose objectstore/{bluestore/{alloc$/{avl} base mem$/{normal-1} onode-segment$/{512K-onoff} write$/{v2/{compr$/{yes$/{lz4}} v2}}}} rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful_host thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} time: 00:18:05 info: https://pulpito.ceph.com/cjames-2026-02-23_16:19:20-rados-main-distro-default-trial/66851/ [66852] rados/thrash-erasure-code/{ceph clusters/{fixed-4} ec_direct_reads/ec_direct_reads_on ec_optimizations/ec_optimizations_on fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore/{alloc$/{avl} base mem$/{normal-2} onode-segment$/{512K-onoff} write$/{v1/{compr$/{yes$/{lz4}} v1}}}} rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects} time: 00:18:03 info: https://pulpito.ceph.com/cjames-2026-02-23_16:19:20-rados-main-distro-default-trial/66852/ [66853] rados/thrash-erasure-code-big/{ceph cluster/{12-osds} ec_direct_reads/ec_direct_reads_on ec_optimizations/ec_optimizations_on mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore/{alloc$/{bitmap} base mem$/{low} onode-segment$/{512K} write$/{v2/{compr$/{no$/{no}} v2}}}} rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:21:22 info: https://pulpito.ceph.com/cjames-2026-02-23_16:19:20-rados-main-distro-default-trial/66853/ 2026-02-23T19:56:05.650 ERROR:teuthology.results:error generating memo/results Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 33, in main results(args['--archive-dir'], args['--name'], args['--email'], File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 78, in results email_results( File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 95, in email_results smtp = smtplib.SMTP('localhost') ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 255, in __init__ (code, msg) = self.connect(host, port) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 341, in connect self.sock = self._get_socket(host, port, self.timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 312, in _get_socket return socket.create_connection((host, port), timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/socket.py", line 115, in create_connection sock.connect(sa) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 590, in connect self._internal_connect(address) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect raise _SocketError(err, strerror(err)) ConnectionRefusedError: [Errno 111] Connection refused