2024-04-02T02:44:40.950 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-02T02:44:40.951 INFO:teuthology.results:seed: '8570' 2024-04-02T02:57:00.715 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-02T02:57:00.718 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-04-02T03:42:04.399 INFO:teuthology.results:Tests finished! gathering results... 2024-04-02T03:42:05.933 INFO:scrape:Found 34 jobs 2024-04-02T03:42:06.169 INFO:scrape:Found 1 distinct failure reasons 2024-04-02T03:42:06.169 INFO:scrape:Failure: Command failed on smithi052 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log' 2024-04-02T03:42:06.169 INFO:scrape:4 jobs: ['7635287', '7635282', '7635302', '7635293'] 2024-04-02T03:42:06.170 INFO:scrape:suites intersection: ['crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore'] 2024-04-02T03:42:06.170 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/perf/{clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'settings/optimized', 'thrashers/default', 'thrashosds-health', 'workloads/fio_4M_rand_read}', 'workloads/fio_4M_rand_write}', 'workloads/radosbench_4K_seq_read}', 'workloads/snaps-few-objects-localized}'] 2024-04-02T03:42:06.170 INFO:scrape: 2024-04-02T03:42:06.171 INFO:teuthology.results:Sending results to yingxin.cheng@intel.com: Test Run: yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 30 Fail ================================================================= [7635282] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} ----------------------------------------------------------------- time: 00:20:52 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635282/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635282/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9dc2186407de48c0a1cec13147974bfa Command failed on smithi171 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7635287] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_write} ----------------------------------------------------------------- time: 00:08:59 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635287/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635287/ sentry: https://sentry.ceph.com/organizations/ceph/?query=569825c4be834e6a9cd9cd1f0a78b0d6 Command failed on smithi052 with status 1: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log' [7635293] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_seq_read} ----------------------------------------------------------------- time: 00:23:12 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635293/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635293/ sentry: https://sentry.ceph.com/organizations/ceph/?query=19b08e7a20a74425a04b1b0eb4e5574a Command failed on smithi037 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7635302] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:35:05 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635302/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635302/ Command failed on smithi069 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' Pass ================================================================= [7635275] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_api_tests} time: 00:27:53 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635275/ [7635276] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} time: 00:14:03 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635276/ [7635277] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} time: 00:46:42 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635277/ [7635278] crimson-rados/singleton/{all/osd-backfill crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/bluestore rados} time: 00:14:53 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635278/ [7635279] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:15:02 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635279/ [7635280] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_rw} time: 00:13:54 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635280/ [7635281] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:29:46 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635281/ [7635283] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} time: 00:19:45 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635283/ [7635284] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:26:43 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635284/ [7635285] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} time: 00:16:37 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635285/ [7635286] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} time: 00:26:23 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635286/ [7635288] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_python} time: 00:16:56 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635288/ [7635289] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} time: 00:12:05 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635289/ [7635290] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:21:02 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635290/ [7635291] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read} time: 00:11:53 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635291/ [7635292] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:22:20 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635292/ [7635294] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} time: 00:12:09 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635294/ [7635295] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} time: 00:13:51 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635295/ [7635296] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} time: 00:21:22 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635296/ [7635297] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} time: 00:12:20 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635297/ [7635298] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:28:30 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635298/ [7635299] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/readwrite} time: 00:12:37 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635299/ [7635300] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests} time: 00:24:22 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635300/ [7635301] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} time: 00:10:32 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635301/ [7635303] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_omap_write} time: 00:24:45 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635303/ [7635304] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:26:31 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635304/ [7635305] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format} time: 00:20:06 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635305/ [7635306] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_fio} time: 00:15:10 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635306/ [7635307] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:14:16 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635307/ [7635308] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} time: 00:10:18 info: https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635308/