2024-03-19T02:38:51.304 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-19T02:38:51.304 INFO:teuthology.results:seed: '8101'
2024-03-19T02:51:59.321 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-19T02:51:59.340 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-03-19T06:22:16.084 INFO:teuthology.results:Tests finished! gathering results...
2024-03-19T06:22:17.723 INFO:scrape:Found 21 jobs
2024-03-19T06:22:19.410 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/sjust-2024-03-19_02:36:51-crimson-rados-wip-sjust-crimson-testing-2024-03-18-distro-default-smithi/7610304/teuthology.log
2024-03-19T06:22:19.813 INFO:scrape:Found 6 distinct failure reasons
2024-03-19T06:22:19.813 INFO:scrape:Timeout 120 running ceph --cluster ceph tell 1.0 deep-scrub
2024-03-19T06:22:19.814 INFO:scrape:11 jobs: ['7610300', '7610302', '7610292', '7610285', '7610288', '7610295', '7610294', '7610287', '7610284', '7610291', '7610298']
2024-03-19T06:22:19.814 INFO:scrape:suites intersection: ['crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph']
2024-03-19T06:22:19.814 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/rbd/{clusters/fixed-1', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'tasks/rbd_python_api_tests}', 'thrashers/default', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/pool-snaps-few-objects}', 'workloads/radosbench-high-concurrency}', 'workloads/radosbench}', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}']
2024-03-19T06:22:19.814 INFO:scrape:
2024-03-19T06:22:19.814 INFO:scrape:Timeout 120 running ceph --cluster ceph tell 2.7 deep-scrub
2024-03-19T06:22:19.814 INFO:scrape:1 jobs: ['7610303']
2024-03-19T06:22:19.814 INFO:scrape:suites: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}']
2024-03-19T06:22:19.814 INFO:scrape:
2024-03-19T06:22:19.814 INFO:scrape:Failure: "2024-03-19T03:28:29.811632+0000 mon.a (mon.0) 204 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 30 sec, mon.a has slow ops (SLOW_OPS)" in cluster log
2024-03-19T06:22:19.815 INFO:scrape:5 jobs: ['7610290', '7610286', '7610282', '7610293', '7610296']
2024-03-19T06:22:19.815 INFO:scrape:suites intersection: ['crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph']
2024-03-19T06:22:19.815 INFO:scrape:suites union: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}', 'tasks/rbd_api_tests}', 'tasks/rbd_cls_tests}', 'tasks/rbd_lock_and_fence}', 'tasks/readwrite}']
2024-03-19T06:22:19.815 INFO:scrape:
2024-03-19T06:22:19.815 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh --crimson
2024-03-19T06:22:19.815 INFO:scrape:1 jobs: ['7610281']
2024-03-19T06:22:19.815 INFO:scrape:suites: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}']
2024-03-19T06:22:19.815 INFO:scrape:
2024-03-19T06:22:19.815 INFO:scrape:Timeout 120 running ceph --cluster ceph pg dump --format=json
2024-03-19T06:22:19.815 INFO:scrape:1 jobs: ['7610289']
2024-03-19T06:22:19.816 INFO:scrape:suites: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_python}']
2024-03-19T06:22:19.816 INFO:scrape:
2024-03-19T06:22:19.816 INFO:scrape:Dead: None
2024-03-19T06:22:19.816 INFO:scrape:1 jobs: ['7610304']
2024-03-19T06:22:19.816 INFO:scrape: