2024-01-16T09:44:08.557 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-16T09:44:08.557 INFO:teuthology.results:seed: '7803' 2024-01-16T10:05:21.067 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-16T10:05:21.088 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-01-16T13:23:33.737 INFO:teuthology.results:Tests finished! gathering results... 2024-01-16T13:23:35.940 INFO:scrape:Found 34 jobs 2024-01-16T13:23:37.396 INFO:scrape:Found 3 distinct failure reasons 2024-01-16T13:23:37.397 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2024-01-16T13:23:37.397 INFO:scrape:1 jobs: ['7518052'] 2024-01-16T13:23:37.397 INFO:scrape:suites: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}'] 2024-01-16T13:23:37.397 INFO:scrape: 2024-01-16T13:23:37.397 INFO:scrape:Timeout 1h running clone.client.0/qa/workunits/rados/test_python.sh -m \ 2024-01-16T13:23:37.397 INFO:scrape:1 jobs: ['7518065'] 2024-01-16T13:23:37.397 INFO:scrape:suites: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_python}'] 2024-01-16T13:23:37.397 INFO:scrape: 2024-01-16T13:23:37.398 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2024-01-16T13:23:37.398 INFO:scrape:2 jobs: ['7518060', '7518054'] 2024-01-16T13:23:37.398 INFO:scrape:suites intersection: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph'] 2024-01-16T13:23:37.398 INFO:scrape:suites union: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}', 'tasks/rbd_api_tests}'] 2024-01-16T13:23:37.398 INFO:scrape: 2024-01-16T13:23:37.399 INFO:teuthology.results:Sending results to yingxin.cheng@intel.com: Test Run: yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 30 Fail ================================================================= [7518052] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_api_tests} ----------------------------------------------------------------- time: 03:29:38 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518052/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518052/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6597a3a741234e95a4c398704d80acfc Command failed (workunit test rados/test.sh) on smithi124 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=95a83278828673203387db3995fc9e411f93e9dd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7518054] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} ----------------------------------------------------------------- time: 03:29:30 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518054/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518054/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a5231e9a0e5747bcbf55ba4fd8720dad Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi194 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=95a83278828673203387db3995fc9e411f93e9dd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crims on_librbd.sh' [7518060] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} ----------------------------------------------------------------- time: 03:29:14 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518060/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518060/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b1bc46d7d14640a4a78880da25532ecb Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi137 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=95a83278828673203387db3995fc9e411f93e9dd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' [7518065] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_python} ----------------------------------------------------------------- time: 01:28:57 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518065/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518065/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b9dfd7bd851c4e24a254d61aa2b6fdbd Command failed (workunit test rados/test_python.sh) on smithi001 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=95a83278828673203387db3995fc9e411f93e9dd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh -m \'not (wait or tier or ec or bench or stats)\'' Pass ================================================================= [7518053] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} time: 00:14:23 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518053/ [7518055] crimson-rados/singleton/{all/osd-backfill crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/bluestore rados} time: 00:13:51 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518055/ [7518056] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:15:06 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518056/ [7518057] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_rw} time: 00:15:08 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518057/ [7518058] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:27:10 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518058/ [7518059] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} time: 00:13:49 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518059/ [7518061] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:25:19 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518061/ [7518062] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} time: 00:14:26 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518062/ [7518063] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} time: 00:26:04 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518063/ [7518064] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_write} time: 00:15:57 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518064/ [7518066] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} time: 00:11:59 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518066/ [7518067] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:20:09 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518067/ [7518068] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read} time: 00:12:14 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518068/ [7518069] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:20:50 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518069/ [7518070] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_seq_read} time: 00:11:24 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518070/ [7518071] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} time: 00:10:07 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518071/ [7518072] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} time: 00:11:47 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518072/ [7518073] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} time: 00:19:36 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518073/ [7518074] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} time: 00:11:26 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518074/ [7518075] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:27:58 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518075/ [7518076] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/readwrite} time: 00:13:10 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518076/ [7518077] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests} time: 00:21:53 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518077/ [7518078] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} time: 00:10:49 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518078/ [7518079] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:26:37 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518079/ [7518080] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_omap_write} time: 00:21:02 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518080/ [7518081] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:29:06 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518081/ [7518082] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format} time: 00:19:27 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518082/ [7518083] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_fio} time: 00:14:32 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518083/ [7518084] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:14:18 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518084/ [7518085] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} time: 00:10:40 info: https://pulpito.ceph.com/yingxin-2024-01-16_07:50:47-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-6-distro-default-smithi/7518085/