2024-03-15T09:30:32.456 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-15T09:30:32.456 INFO:teuthology.results:seed: '3' 2024-03-15T09:57:17.402 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-15T09:57:17.406 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-15T12:54:30.834 INFO:teuthology.results:Tests finished! gathering results... 2024-03-15T12:54:33.633 INFO:scrape:Found 34 jobs 2024-03-15T12:54:34.835 INFO:scrape:Found 3 distinct failure reasons 2024-03-15T12:54:34.835 INFO:scrape:Failure: "2024-03-15T10:20:03.449543+0000 mon.a (mon.0) 743 : cluster [ERR] Health check failed: 9 scrub errors (OSD_SCRUB_ERRORS)" in cluster log 2024-03-15T12:54:34.835 INFO:scrape:6 jobs: ['7601902', '7601907', '7601885', '7601891', '7601901', '7601896'] 2024-03-15T12:54:34.835 INFO:scrape:suites intersection: ['crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph'] 2024-03-15T12:54:34.836 INFO:scrape:suites union: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}', 'tasks/rbd_cls_tests}', 'tasks/rbd_lock_and_fence}', 'tasks/rbd_python_api_tests_old_format}', 'tasks/rbd_python_api_tests}', 'tasks/readwrite}'] 2024-03-15T12:54:34.836 INFO:scrape: 2024-03-15T12:54:34.836 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh --crimson 2024-03-15T12:54:34.836 INFO:scrape:1 jobs: ['7601877'] 2024-03-15T12:54:34.836 INFO:scrape:suites: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}'] 2024-03-15T12:54:34.836 INFO:scrape: 2024-03-15T12:54:34.836 INFO:scrape:Timeout 120 running ceph --cluster ceph pg dump --format=json 2024-03-15T12:54:34.836 INFO:scrape:1 jobs: ['7601890'] 2024-03-15T12:54:34.836 INFO:scrape:suites: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_python}'] 2024-03-15T12:54:34.837 INFO:scrape: 2024-03-15T12:54:34.838 INFO:teuthology.results:Sending results to yingxin.cheng@intel.com: Test Run: yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/ failed: 8 dead: 0 running: 0 waiting: 0 queued: 0 passed: 26 Fail ================================================================= [7601877] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_api_tests} ----------------------------------------------------------------- time: 03:12:06 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601877/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601877/ sentry: https://sentry.ceph.com/organizations/ceph/?query=79d89718e82b4e639760c7e363e1848e Command failed (workunit test rados/test.sh) on smithi090 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1291d4cf505875aeee31208758583eedb6ffa353 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson' [7601885] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} ----------------------------------------------------------------- time: 00:16:51 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601885/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601885/ "2024-03-15T10:04:05.116421+0000 mon.a (mon.0) 440 : cluster [ERR] Health check failed: 9 scrub errors (OSD_SCRUB_ERRORS)" in cluster log [7601890] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_python} ----------------------------------------------------------------- time: 00:19:50 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601890/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601890/ Command failed on smithi008 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' [7601891] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} ----------------------------------------------------------------- time: 00:11:54 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601891/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601891/ "2024-03-15T10:01:00.022959+0000 mon.a (mon.0) 205 : cluster [ERR] Health check failed: 6 scrub errors (OSD_SCRUB_ERRORS)" in cluster log [7601896] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} ----------------------------------------------------------------- time: 00:10:44 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601896/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601896/ "2024-03-15T10:02:43.676577+0000 mon.a (mon.0) 143 : cluster [ERR] Health check failed: 9 scrub errors (OSD_SCRUB_ERRORS)" in cluster log [7601901] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/readwrite} ----------------------------------------------------------------- time: 00:14:56 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601901/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601901/ "2024-03-15T10:10:03.740503+0000 mon.a (mon.0) 165 : cluster [ERR] Health check failed: 9 scrub errors (OSD_SCRUB_ERRORS)" in cluster log [7601902] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests} ----------------------------------------------------------------- time: 00:21:15 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601902/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601902/ "2024-03-15T10:20:03.449543+0000 mon.a (mon.0) 743 : cluster [ERR] Health check failed: 9 scrub errors (OSD_SCRUB_ERRORS)" in cluster log [7601907] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format} ----------------------------------------------------------------- time: 00:19:34 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601907/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601907/ "2024-03-15T10:21:38.398446+0000 mon.a (mon.0) 648 : cluster [ERR] Health check failed: 7 scrub errors (OSD_SCRUB_ERRORS)" in cluster log Pass ================================================================= [7601878] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} time: 00:13:33 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601878/ [7601879] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} time: 00:41:59 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601879/ [7601880] crimson-rados/singleton/{all/osd-backfill crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/bluestore rados} time: 00:14:35 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601880/ [7601881] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:16:39 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601881/ [7601882] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_rw} time: 00:14:11 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601882/ [7601883] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:26:46 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601883/ [7601884] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} time: 00:12:45 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601884/ [7601886] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:25:12 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601886/ [7601887] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} time: 00:13:38 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601887/ [7601888] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} time: 00:28:33 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601888/ [7601889] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_write} time: 00:13:36 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601889/ [7601892] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:21:01 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601892/ [7601893] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read} time: 00:12:21 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601893/ [7601894] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:19:56 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601894/ [7601895] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_seq_read} time: 00:11:37 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601895/ [7601897] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} time: 00:12:15 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601897/ [7601898] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} time: 00:20:08 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601898/ [7601899] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} time: 00:11:14 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601899/ [7601900] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:28:27 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601900/ [7601903] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} time: 00:11:10 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601903/ [7601904] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:28:57 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601904/ [7601905] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_omap_write} time: 00:24:30 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601905/ [7601906] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:26:56 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601906/ [7601908] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_fio} time: 00:14:44 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601908/ [7601909] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:15:01 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601909/ [7601910] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} time: 00:10:52 info: https://pulpito.ceph.com/yingxin-2024-03-15_08:49:37-crimson-rados-ci-yingxin-baseline-distro-default-smithi/7601910/