2024-04-01T01:20:34.727 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-01T01:20:34.728 INFO:teuthology.results:seed: '348' 2024-04-01T01:36:11.905 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-01T01:36:11.909 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-04-01T02:19:15.360 INFO:teuthology.results:Tests finished! gathering results... 2024-04-01T02:19:18.581 INFO:scrape:Found 34 jobs 2024-04-01T02:19:18.819 INFO:scrape:Found 1 distinct failure reasons 2024-04-01T02:19:18.819 INFO:scrape:Timeout 120 running ceph --cluster ceph pg dump --format=json 2024-04-01T02:19:18.819 INFO:scrape:1 jobs: ['7633777'] 2024-04-01T02:19:18.819 INFO:scrape:suites: ['crimson-rados/perf/{clusters/{fixed-2}', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'settings/optimized', 'workloads/radosbench_4K_seq_read}'] 2024-04-01T02:19:18.819 INFO:scrape: 2024-04-01T02:19:18.821 INFO:teuthology.results:Sending results to yingxin.cheng@intel.com: Test Run: yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/ failed: 1 dead: 0 running: 0 waiting: 0 queued: 0 passed: 33 Fail ================================================================= [7633777] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_seq_read} ----------------------------------------------------------------- time: 00:19:56 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633777/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633777/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9843dc361a1944d3a2c27792e66de796 Command failed on smithi174 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' Pass ================================================================= [7633759] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_api_tests} time: 00:29:36 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633759/ [7633760] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} time: 00:14:56 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633760/ [7633761] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} time: 00:48:23 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633761/ [7633762] crimson-rados/singleton/{all/osd-backfill crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/bluestore rados} time: 00:15:08 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633762/ [7633763] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:16:27 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633763/ [7633764] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_rw} time: 00:15:48 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633764/ [7633765] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:28:49 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633765/ [7633766] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} time: 00:14:31 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633766/ [7633767] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} time: 00:20:22 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633767/ [7633768] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:25:26 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633768/ [7633769] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} time: 00:14:11 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633769/ [7633770] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} time: 00:27:23 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633770/ [7633771] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_write} time: 00:15:50 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633771/ [7633772] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_python} time: 00:14:14 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633772/ [7633773] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} time: 00:12:51 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633773/ [7633774] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:19:25 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633774/ [7633775] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read} time: 00:11:26 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633775/ [7633776] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:21:13 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633776/ [7633778] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} time: 00:10:13 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633778/ [7633779] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} time: 00:13:49 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633779/ [7633780] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} time: 00:19:19 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633780/ [7633781] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} time: 00:11:08 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633781/ [7633782] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:28:01 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633782/ [7633783] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/readwrite} time: 00:12:25 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633783/ [7633784] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests} time: 00:24:21 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633784/ [7633785] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} time: 00:10:42 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633785/ [7633786] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:27:11 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633786/ [7633787] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_omap_write} time: 00:23:00 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633787/ [7633788] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:27:23 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633788/ [7633789] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format} time: 00:20:37 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633789/ [7633790] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_fio} time: 00:14:07 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633790/ [7633791] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:14:12 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633791/ [7633792] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} time: 00:10:41 info: https://pulpito.ceph.com/yingxin-2024-04-01_01:16:44-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7633792/