2024-03-09T06:33:48.045 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-09T06:33:48.045 INFO:teuthology.results:subset: '111/120000' 2024-03-09T06:33:48.045 INFO:teuthology.results:seed: '279' 2024-03-09T06:54:53.024 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-09T06:54:53.029 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-09T08:09:59.004 INFO:teuthology.results:Tests finished! gathering results... 2024-03-09T08:10:01.712 INFO:scrape:Found 34 jobs 2024-03-09T08:10:02.204 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586654/teuthology.log 2024-03-09T08:10:02.773 INFO:scrape:Found 3 distinct failure reasons 2024-03-09T08:10:02.774 INFO:scrape:Failure: reached maximum tries (501) after waiting for 3000 seconds 2024-03-09T08:10:02.774 INFO:scrape:1 jobs: ['7586635'] 2024-03-09T08:10:02.774 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/radosbench-high-concurrency}'] 2024-03-09T08:10:02.774 INFO:scrape: 2024-03-09T08:10:02.774 INFO:scrape:Dead: None 2024-03-09T08:10:02.774 INFO:scrape:1 jobs: ['7586654'] 2024-03-09T08:10:02.774 INFO:scrape: 2024-03-09T08:10:02.775 INFO:scrape:Failure: Command failed on smithi112 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats' 2024-03-09T08:10:02.775 INFO:scrape:1 jobs: ['7586643'] 2024-03-09T08:10:02.775 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-localized}'] 2024-03-09T08:10:02.775 INFO:scrape: 2024-03-09T08:10:02.776 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/ failed: 2 dead: 1 running: 0 waiting: 0 queued: 0 passed: 31 Fail ================================================================= [7586635] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 01:19:21 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586635/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586635/ sentry: https://sentry.ceph.com/organizations/ceph/?query=884cbf589a39425eb056402a65180d8a reached maximum tries (501) after waiting for 3000 seconds [7586643] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:30:04 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586643/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586643/ Command failed on smithi112 with status 6: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats' Dead ================================================================= [7586654] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_omap_write} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586654/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586654/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds Pass ================================================================= [7586626] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_api_tests} time: 00:43:00 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586626/ [7586627] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} time: 00:15:29 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586627/ [7586628] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} time: 00:42:03 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586628/ [7586629] crimson-rados/singleton/{all/osd-backfill crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/bluestore rados} time: 00:15:23 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586629/ [7586630] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:20:01 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586630/ [7586631] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_rw} time: 00:22:36 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586631/ [7586632] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:26:54 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586632/ [7586633] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} time: 00:15:11 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586633/ [7586634] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} time: 00:17:30 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586634/ [7586636] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} time: 00:16:02 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586636/ [7586637] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} time: 00:32:28 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586637/ [7586638] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_write} time: 00:16:23 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586638/ [7586639] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_python} time: 00:13:40 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586639/ [7586640] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} time: 00:11:17 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586640/ [7586641] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:20:52 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586641/ [7586642] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read} time: 00:12:58 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586642/ [7586644] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_seq_read} time: 00:13:17 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586644/ [7586645] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} time: 00:15:48 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586645/ [7586646] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} time: 00:19:10 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586646/ [7586647] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} time: 00:21:48 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586647/ [7586648] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} time: 00:12:26 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586648/ [7586649] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:32:31 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586649/ [7586650] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/readwrite} time: 00:13:10 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586650/ [7586651] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests} time: 00:24:15 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586651/ [7586652] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} time: 00:13:27 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586652/ [7586653] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:26:32 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586653/ [7586655] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:28:16 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586655/ [7586656] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format} time: 00:20:14 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586656/ [7586657] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_fio} time: 00:17:17 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586657/ [7586658] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:15:43 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586658/ [7586659] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} time: 00:11:15 info: https://pulpito.ceph.com/yuriw-2024-03-08_15:52:58-crimson-rados-squid-distro-default-smithi/7586659/