Found 21 jobs Found 5 distinct failure reasons Failure: Command failed on smithi129 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 2.3 deep-scrub' 12 jobs: ['7583511', '7583502', '7583507', '7583504', '7583499', '7583498', '7583516', '7583506', '7583514', '7583500', '7583513', '7583509'] suites intersection: ['crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/basic/{clusters/fixed-2', 'crimson-rados/singleton/{all/osd-backfill', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'rados}', 'tasks/rados_python}', 'tasks/readwrite}', 'thrashers/default', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/pool-snaps-few-objects}', 'workloads/radosbench-high-concurrency}', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}', 'workloads/write_fadvise_dontneed}'] Failure: reached maximum tries (501) after waiting for 3000 seconds 1 jobs: ['7583503'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/radosbench}'] Timeout 3h running clone.client.0/qa/workunits/rados/test.sh --crimson 1 jobs: ['7583496'] suites: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}'] Dead: b'2024-03-07T04:07:43.632 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-07T04:07:43.627+0000 7f4a10eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson' 1 jobs: ['7583510'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects-balanced}'] Failure: Command failed on smithi181 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ea9a5688d9980e4ddedb59679d2960bcbac6fc71' 1 jobs: ['7583497'] suites: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests}']