2024-01-05T09:34:42.213 INFO:root:teuthology version: 0.0.1.dev228+g01dd171 2024-01-05T09:34:42.213 INFO:teuthology.results:seed: '4348' 2024-01-05T10:17:30.507 INFO:root:teuthology version: 0.0.1.dev228+g01dd171 2024-01-05T10:17:30.527 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-01-05T12:58:42.577 INFO:teuthology.results:Tests finished! gathering results... 2024-01-05T12:58:43.494 INFO:scrape:Found 34 jobs 2024-01-05T12:58:43.631 INFO:scrape:Found 2 distinct failure reasons 2024-01-05T12:58:43.632 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2024-01-05T12:58:43.632 INFO:scrape:1 jobs: ['7507195'] 2024-01-05T12:58:43.632 INFO:scrape:suites: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}'] 2024-01-05T12:58:43.632 INFO:scrape: 2024-01-05T12:58:43.632 INFO:scrape:Timeout 120 running ceph --cluster ceph pg dump --format=json 2024-01-05T12:58:43.632 INFO:scrape:1 jobs: ['7507218'] 2024-01-05T12:58:43.632 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects-balanced}'] 2024-01-05T12:58:43.632 INFO:scrape: 2024-01-05T12:58:43.634 INFO:teuthology.results:Sending results to rongqi.sun777@gmail.com: Test Run: svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 32 Fail ================================================================= [7507195] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_api_tests} ----------------------------------------------------------------- time: 03:13:16 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507195/ log: http://qa-proxy.ceph.com/teuthology/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507195/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4b0edb9c034d4f13922acd1935d18260 Command failed (workunit test rados/test.sh) on smithi026 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7ff29cc6bb464521b385ec34b711abefc530d52c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7507218] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:21:56 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507218/ log: http://qa-proxy.ceph.com/teuthology/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507218/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b860a019129048d8ba24d8f1e8fc6cb3 Command failed on smithi118 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' Pass ================================================================= [7507196] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} time: 00:17:02 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507196/ [7507197] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} time: 00:49:49 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507197/ [7507198] crimson-rados/singleton/{all/osd-backfill crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/bluestore rados} time: 00:15:20 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507198/ [7507199] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:16:40 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507199/ [7507200] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_rw} time: 00:17:46 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507200/ [7507201] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:27:58 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507201/ [7507202] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} time: 00:15:29 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507202/ [7507203] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} time: 00:19:01 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507203/ [7507204] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:25:26 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507204/ [7507205] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} time: 00:16:07 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507205/ [7507206] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} time: 00:26:56 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507206/ [7507207] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_write} time: 00:15:52 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507207/ [7507208] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_python} time: 00:16:58 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507208/ [7507209] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} time: 00:13:03 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507209/ [7507210] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:22:02 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507210/ [7507211] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read} time: 00:13:18 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507211/ [7507212] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:21:01 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507212/ [7507213] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_seq_read} time: 00:12:53 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507213/ [7507214] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} time: 00:11:31 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507214/ [7507215] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} time: 00:13:27 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507215/ [7507216] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} time: 00:20:41 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507216/ [7507217] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} time: 00:13:27 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507217/ [7507219] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/readwrite} time: 00:14:13 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507219/ [7507220] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests} time: 00:23:00 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507220/ [7507221] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} time: 00:12:13 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507221/ [7507222] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:27:57 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507222/ [7507223] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_omap_write} time: 00:22:47 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507223/ [7507224] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:27:47 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507224/ [7507225] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format} time: 00:20:28 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507225/ [7507226] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_fio} time: 00:16:18 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507226/ [7507227] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:15:00 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507227/ [7507228] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} time: 00:12:29 info: https://pulpito.ceph.com/svelar-2024-01-04_02:49:24-crimson-rados-wip-svelar-testing-crimson-2024-01-03-0828-distro-default-smithi/7507228/