2026-02-23T09:59:38.343 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0
2026-02-23T09:59:38.343 INFO:teuthology.results:seed: '66'
2026-02-23T11:44:41.062 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0
2026-02-23T11:44:41.070 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2026-02-23T13:18:53.671 INFO:teuthology.results:Tests finished! gathering results...
2026-02-23T13:18:59.411 INFO:scrape:Found 280 jobs
2026-02-23T13:19:07.899 INFO:scrape:Found 7 distinct failure reasons
2026-02-23T13:19:07.899 INFO:scrape:Failure: "grep: /var/log/ceph/0e40012e-10a5-11f1-b261-d404e6e7d460/ceph.log: No such file or directory" in cluster log
2026-02-23T13:19:07.899 INFO:scrape:39 jobs
2026-02-23T13:19:07.899 INFO:scrape:suites intersection: ['clusters/{crimson-fixed-1}', 'crimson-rados/perf/{0-crimson_install', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/cephadm', 'perf_ignorelist', 'settings/optimized']
2026-02-23T13:19:07.899 INFO:scrape:suites union: ['clusters/{crimson-fixed-1}', 'crimson-rados/perf/{0-crimson_install', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/cephadm', 'objectstore/crimson_bluestore', 'objectstore/seastore/rbm$/{crimson_seastore_rbm_2q}', 'objectstore/seastore/rbm$/{crimson_seastore_rbm}', 'objectstore/seastore/segmented$/{crimson_seastore_segmented_2q}', 'objectstore/seastore/segmented$/{crimson_seastore_segmented}', 'perf_ignorelist', 'settings/optimized', 'workloads/fio_4K_rand_read}', 'workloads/fio_4K_rand_rw}', 'workloads/fio_4M_rand_read}', 'workloads/fio_4M_rand_rw}', 'workloads/fio_4M_rand_write}', 'workloads/radosbench_4K_rand_read}', 'workloads/radosbench_4K_seq_read}', 'workloads/radosbench_4M_rand_read}', 'workloads/radosbench_4M_seq_read}', 'workloads/radosbench_4M_write}', 'workloads/radosbench_omap_write}', 'workloads/sample_fio}', 'workloads/sample_radosbench}']
2026-02-23T13:19:07.899 INFO:scrape:
2026-02-23T13:19:07.899 INFO:scrape:Failure: no mgrs on the same host as first mon a
2026-02-23T13:19:07.899 INFO:scrape:78 jobs
2026-02-23T13:19:07.900 INFO:scrape:suites intersection: ['crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/cephadm']
2026-02-23T13:19:07.900 INFO:scrape:suites union: ['0-size-min-size-overrides/3-size-2-min-size', '1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', 'clusters/crimson-fixed-2', 'clusters/{crimson-fixed-2}', 'crimson-rados/basic/{0-crimson_install', 'crimson-rados/thrash/{0-crimson_install', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/cephadm', 'objectstore/crimson_bluestore', 'objectstore/seastore/rbm$/{crimson_seastore_rbm_2q}', 'objectstore/seastore/rbm$/{crimson_seastore_rbm}', 'objectstore/seastore/segmented$/{crimson_seastore_segmented_2q}', 'objectstore/seastore/segmented$/{crimson_seastore_segmented}', 'tasks/cls}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'tasks/readwrite}', 'thrashers/{default}', 'thrashers/{pggrow}', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/pool-snaps-few-objects}', 'workloads/radosbench-high-concurrency}', 'workloads/radosbench}', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}', 'workloads/write_fadvise_dontneed}']
2026-02-23T13:19:07.900 INFO:scrape:
2026-02-23T13:19:07.900 INFO:scrape:Failure: "2026-02-23T10:40:52.084231+0000 mgr.x (mgr.14152) 60 : cluster [ERR] Unhandled exception from module 'devicehealth' while running on mgr.x: osd pool create failed: set-allow-crimson must be set to create a pool with the crimson flag (--crimson specified or osd_pool_default_crimson set) retval: -22" in cluster log
2026-02-23T13:19:07.900 INFO:scrape:18 jobs: ['66099', '66199', '66130', '66169', '66292', '66114', '66215', '66277', '66068', '66262', '66052', '66231', '66245', '66184', '66145', '66161', '66083', '66037']
2026-02-23T13:19:07.900 INFO:scrape:suites intersection: ['clusters/crimson-fixed-1', 'crimson-rados/rbd/{0-crimson_install', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/cephadm']
2026-02-23T13:19:07.900 INFO:scrape:suites union: ['clusters/crimson-fixed-1', 'crimson-rados/rbd/{0-crimson_install', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/cephadm', 'objectstore/crimson_bluestore', 'objectstore/seastore/rbm$/{crimson_seastore_rbm_2q}', 'objectstore/seastore/rbm$/{crimson_seastore_rbm}', 'objectstore/seastore/segmented$/{crimson_seastore_segmented_2q}', 'objectstore/seastore/segmented$/{crimson_seastore_segmented}', 'tasks/rbd_api_tests_old_format}', 'tasks/rbd_api_tests}', 'tasks/rbd_cls_tests}', 'tasks/rbd_lock_and_fence}', 'tasks/rbd_python_api_tests_old_format}', 'tasks/rbd_python_api_tests}']
2026-02-23T13:19:07.900 INFO:scrape:
2026-02-23T13:19:07.900 INFO:scrape:Timeout 120 running ceph --cluster ceph osd dump --format=json
2026-02-23T13:19:07.900 INFO:scrape:1 jobs: ['66217']
2026-02-23T13:19:07.900 INFO:scrape:suites: ['0-size-min-size-overrides/3-size-2-min-size', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', 'clusters/{crimson-fixed-2}', 'crimson-rados/thrash/{0-crimson_install', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/seastore/rbm$/{crimson_seastore_rbm_2q}', 'thrashers/{pggrow}', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}']
2026-02-23T13:19:07.900 INFO:scrape:
2026-02-23T13:19:07.900 INFO:scrape:Failure: failed to complete snap trimming before timeout
2026-02-23T13:19:07.900 INFO:scrape:1 jobs: ['66202']
2026-02-23T13:19:07.900 INFO:scrape:suites: ['0-size-min-size-overrides/3-size-2-min-size', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', 'clusters/{crimson-fixed-2}', 'crimson-rados/thrash/{0-crimson_install', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/seastore/rbm$/{crimson_seastore_rbm_2q}', 'thrashers/{pggrow}', 'thrashosds-health', 'workloads/small-objects-localized}']
2026-02-23T13:19:07.900 INFO:scrape:
2026-02-23T13:19:07.900 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on trial131 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e371433fcd1390a0b82cfb18effc422c1f5d8c1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson'
2026-02-23T13:19:07.900 INFO:scrape:1 jobs: ['66273']
2026-02-23T13:19:07.900 INFO:scrape:suites: ['clusters/crimson-fixed-2', 'crimson-rados/basic/{0-crimson_install', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/crimson_bluestore', 'tasks/rados_api_tests}']
2026-02-23T13:19:07.900 INFO:scrape:
2026-02-23T13:19:07.900 INFO:scrape:Failure: reached maximum tries (500) after waiting for 3000 seconds
2026-02-23T13:19:07.900 INFO:scrape:2 jobs: ['66080', '66263']
2026-02-23T13:19:07.900 INFO:scrape:suites intersection: ['0-size-min-size-overrides/3-size-2-min-size', '2-recovery-overrides/{default}', 'clusters/{crimson-fixed-2}', 'crimson-rados/thrash/{0-crimson_install', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'thrashosds-health']
2026-02-23T13:19:07.901 INFO:scrape:suites union: ['0-size-min-size-overrides/3-size-2-min-size', '1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', 'clusters/{crimson-fixed-2}', 'crimson-rados/thrash/{0-crimson_install', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/crimson_bluestore', 'objectstore/seastore/segmented$/{crimson_seastore_segmented_2q}', 'thrashers/{default}', 'thrashers/{pggrow}', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/radosbench-high-concurrency}']
2026-02-23T13:19:07.901 INFO:scrape: