2023-10-16T01:30:52.570 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-16T01:30:52.571 INFO:teuthology.results:seed: '7309' 2023-10-16T01:56:30.441 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-16T01:56:30.445 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-16T13:57:25.479 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-10-16T13:57:27.736 INFO:scrape:Found 34 jobs 2023-10-16T13:57:28.333 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-6685-gb1e0adfe (b1e0adfe7701a3d2e81ca4aa366e9cbe17529fd6) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fd7dfc0e913] , 2: ceph_test_rados(+0x1d2b3) [0x5580c15ca2b3] 2023-10-16T13:57:28.334 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2023-10-16T13:57:28.381 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-6685-gb1e0adfe (b1e0adfe7701a3d2e81ca4aa366e9cbe17529fd6) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fd7dfc0e913] , 2: ceph_test_rados(+0x1d2b3) [0x5580c15ca2b3] 2023-10-16T13:57:28.382 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2023-10-16T13:57:28.436 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-6685-gb1e0adfe (b1e0adfe7701a3d2e81ca4aa366e9cbe17529fd6) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fd7dfc0e913] , 2: ceph_test_rados(+0x1d2b3) [0x5580c15ca2b3] 2023-10-16T13:57:28.437 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2023-10-16T13:57:28.479 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-6685-gb1e0adfe (b1e0adfe7701a3d2e81ca4aa366e9cbe17529fd6) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fd7dfc0e913] , 2: ceph_test_rados(+0x1d2b3) [0x5580c15ca2b3] 2023-10-16T13:57:28.480 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2023-10-16T13:57:30.425 INFO:scrape:Found 10 distinct failure reasons 2023-10-16T13:57:30.426 INFO:scrape:Dead: b'2023-10-16T13:52:40.634 DEBUG:teuthology.exit:Finished running handlers' 2023-10-16T13:57:30.426 INFO:scrape:7 jobs: ['7428845', '7428847', '7428848', '7428851', '7428839', '7428841', '7428830'] 2023-10-16T13:57:30.426 INFO:scrape:suites intersection: ['crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph'] 2023-10-16T13:57:30.427 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/basic/{clusters/fixed-2', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'tasks/readwrite}', 'thrashers/default', 'thrashosds-health', 'workloads/pool-snaps-few-objects}', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}'] 2023-10-16T13:57:30.427 INFO:scrape: 2023-10-16T13:57:30.428 INFO:scrape:Failure: "2023-10-16T02:04:40.288838+0000 mon.a (mon.0) 152 : cluster [WRN] Health check failed: Degraded data redundancy: 26/1596 objects degraded (1.629%), 8 pgs degraded (PG_DEGRADED)" in cluster log 2023-10-16T13:57:30.428 INFO:scrape:7 jobs: ['7428844', '7428842', '7428857', '7428846', '7428850', '7428840', '7428836'] 2023-10-16T13:57:30.429 INFO:scrape:suites intersection: ['crimson-rados/perf/{clusters/{fixed-2}', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'settings/optimized'] 2023-10-16T13:57:30.429 INFO:scrape:suites union: ['crimson-rados/perf/{clusters/{fixed-2}', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'settings/optimized', 'workloads/fio_4M_rand_write}', 'workloads/radosbench_4K_rand_read}', 'workloads/radosbench_4K_seq_read}', 'workloads/radosbench_4M_rand_read}', 'workloads/radosbench_4M_seq_read}', 'workloads/radosbench_4M_write}', 'workloads/sample_radosbench}'] 2023-10-16T13:57:30.430 INFO:scrape: 2023-10-16T13:57:30.430 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' 2023-10-16T13:57:30.430 INFO:scrape:1 jobs: ['7428856'] 2023-10-16T13:57:30.431 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/write_fadvise_dontneed}'] 2023-10-16T13:57:30.431 INFO:scrape: 2023-10-16T13:57:30.432 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd_python.sh -m \ 2023-10-16T13:57:30.432 INFO:scrape:2 jobs: ['7428854', '7428849'] 2023-10-16T13:57:30.432 INFO:scrape:suites intersection: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph'] 2023-10-16T13:57:30.433 INFO:scrape:suites union: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}', 'tasks/rbd_python_api_tests}'] 2023-10-16T13:57:30.433 INFO:scrape: 2023-10-16T13:57:30.433 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2023-10-16T13:57:30.434 INFO:scrape:2 jobs: ['7428826', '7428832'] 2023-10-16T13:57:30.434 INFO:scrape:suites intersection: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph'] 2023-10-16T13:57:30.435 INFO:scrape:suites union: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}', 'tasks/rbd_api_tests}'] 2023-10-16T13:57:30.435 INFO:scrape: 2023-10-16T13:57:30.435 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2023-10-16T13:57:30.436 INFO:scrape:1 jobs: ['7428838'] 2023-10-16T13:57:30.436 INFO:scrape:suites: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_cls_tests}'] 2023-10-16T13:57:30.437 INFO:scrape: 2023-10-16T13:57:30.437 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2023-10-16T13:57:30.437 INFO:scrape:1 jobs: ['7428824'] 2023-10-16T13:57:30.438 INFO:scrape:suites: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}'] 2023-10-16T13:57:30.438 INFO:scrape: 2023-10-16T13:57:30.438 INFO:scrape:Timeout 1h running clone.client.0/qa/workunits/rados/test_python.sh -m \ 2023-10-16T13:57:30.439 INFO:scrape:1 jobs: ['7428837'] 2023-10-16T13:57:30.439 INFO:scrape:suites: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_python}'] 2023-10-16T13:57:30.440 INFO:scrape: 2023-10-16T13:57:30.440 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2023-10-16T13:57:30.440 INFO:scrape:1 jobs: ['7428843'] 2023-10-16T13:57:30.441 INFO:scrape:suites: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_lock_and_fence}'] 2023-10-16T13:57:30.441 INFO:scrape: 2023-10-16T13:57:30.442 INFO:scrape:Dead: b'2023-10-16T13:56:43.474 INFO:teuthology.orchestra.run.smithi179.stderr:2023-10-16T13:56:43.470+0000 7f55cc59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson' 2023-10-16T13:57:30.442 INFO:scrape:1 jobs: ['7428853'] 2023-10-16T13:57:30.442 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects}'] 2023-10-16T13:57:30.443 INFO:scrape: 2023-10-16T13:57:30.445 INFO:teuthology.results:Sending results to yingxin.cheng@intel.com: Test Run: yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/ failed: 16 dead: 6 running: 2 waiting: 0 queued: 0 passed: 10 Fail ================================================================= [7428824] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_api_tests} ----------------------------------------------------------------- time: 03:13:04 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428824/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428824/ sentry: https://sentry.ceph.com/organizations/ceph/?query=da95527c756c4c2ba7b24b9e81534c1e Command failed (workunit test rados/test.sh) on smithi040 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b1e0adfe7701a3d2e81ca4aa366e9cbe17529fd6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7428826] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} ----------------------------------------------------------------- time: 03:12:31 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428826/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428826/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cdf1d7c649e446d9b5b01c3e5841f409 Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi138 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b1e0adfe7701a3d2e81ca4aa366e9cbe17529fd6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crims on_librbd.sh' [7428832] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} ----------------------------------------------------------------- time: 03:12:28 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428832/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428832/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f344c6b1340c4dcc9effdc51d428f37d Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi129 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b1e0adfe7701a3d2e81ca4aa366e9cbe17529fd6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' [7428836] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_write} ----------------------------------------------------------------- time: 00:15:50 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428836/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428836/ "2023-10-16T01:56:05.373977+0000 mon.a (mon.0) 127 : cluster [WRN] Health check failed: Reduced data availability: 7 pgs inactive (PG_AVAILABILITY)" in cluster log [7428837] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_python} ----------------------------------------------------------------- time: 01:11:34 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428837/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428837/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fc2549c96d1647efa5b591117391e149 Command failed (workunit test rados/test_python.sh) on smithi078 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b1e0adfe7701a3d2e81ca4aa366e9cbe17529fd6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh -m \'not (wait or tier or ec or bench or stats)\'' [7428838] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} ----------------------------------------------------------------- time: 03:12:08 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428838/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428838/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a95df37ff1c43189dc725e8f9b403b1 Command failed (workunit test cls/test_cls_rbd.sh) on smithi114 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b1e0adfe7701a3d2e81ca4aa366e9cbe17529fd6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7428840] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read} ----------------------------------------------------------------- time: 00:12:47 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428840/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428840/ "2023-10-16T01:59:56.118958+0000 mon.a (mon.0) 156 : cluster [WRN] Health check failed: Degraded data redundancy: 145/9244 objects degraded (1.569%), 10 pgs degraded (PG_DEGRADED)" in cluster log [7428842] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_seq_read} ----------------------------------------------------------------- time: 00:12:37 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428842/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428842/ "2023-10-16T02:04:40.605097+0000 mon.a (mon.0) 151 : cluster [WRN] Health check failed: Degraded data redundancy: 73/8886 objects degraded (0.822%), 6 pgs degraded (PG_DEGRADED)" in cluster log [7428843] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} ----------------------------------------------------------------- time: 03:12:37 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428843/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428843/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1d3b7bbc94774e57861da4fde47efd13 Command failed (workunit test rbd/test_lock_fence.sh) on smithi032 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b1e0adfe7701a3d2e81ca4aa366e9cbe17529fd6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh' [7428844] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} ----------------------------------------------------------------- time: 00:13:14 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428844/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428844/ "2023-10-16T02:04:40.288838+0000 mon.a (mon.0) 152 : cluster [WRN] Health check failed: Degraded data redundancy: 26/1596 objects degraded (1.629%), 8 pgs degraded (PG_DEGRADED)" in cluster log [7428846] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} ----------------------------------------------------------------- time: 00:12:29 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428846/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428846/ "2023-10-16T02:06:26.973433+0000 mon.a (mon.0) 153 : cluster [WRN] Health check failed: Degraded data redundancy: 34/1756 objects degraded (1.936%), 9 pgs degraded (PG_DEGRADED)" in cluster log [7428849] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests} ----------------------------------------------------------------- time: 03:11:05 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428849/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428849/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2a0614daa66d4f998404d765ff25c986 Command failed (workunit test rbd/test_librbd_python.sh) on smithi124 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b1e0adfe7701a3d2e81ca4aa366e9cbe17529fd6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh -m \'not skip_if_crimson\'' [7428850] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} ----------------------------------------------------------------- time: 00:11:53 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428850/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428850/ "2023-10-16T02:08:17.120835+0000 mon.a (mon.0) 153 : cluster [WRN] Health check failed: Degraded data redundancy: 24/1684 objects degraded (1.425%), 2 pgs degraded (PG_DEGRADED)" in cluster log [7428854] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format} ----------------------------------------------------------------- time: 03:11:15 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428854/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428854/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4212a6a8230a4958a0d49d21fb0ca7c8 Command failed (workunit test rbd/test_librbd_python.sh) on smithi175 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b1e0adfe7701a3d2e81ca4aa366e9cbe17529fd6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh -m \'not skip_if_crimson\'' [7428856] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} ----------------------------------------------------------------- time: 00:23:05 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428856/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428856/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --write-fadvise- dontneed --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' [7428857] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} ----------------------------------------------------------------- time: 00:11:52 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428857/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428857/ "2023-10-16T02:15:28.726596+0000 mon.a (mon.0) 152 : cluster [WRN] Health check failed: Degraded data redundancy: 114/10702 objects degraded (1.065%), 6 pgs degraded (PG_DEGRADED)" in cluster log Dead ================================================================= [7428830] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428830/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428830/ hit max job timeout [7428839] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428839/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428839/ hit max job timeout [7428841] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428841/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428841/ hit max job timeout [7428845] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428845/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428845/ hit max job timeout [7428847] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428847/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428847/ hit max job timeout [7428848] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/readwrite} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428848/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428848/ hit max job timeout Running ================================================================= [7428851] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-localized} info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428851/ [7428853] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects} info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428853/ Pass ================================================================= [7428825] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} time: 00:15:43 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428825/ [7428827] crimson-rados/singleton/{all/osd-backfill crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/bluestore rados} time: 00:15:47 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428827/ [7428828] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:15:59 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428828/ [7428829] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_rw} time: 00:16:39 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428829/ [7428831] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} time: 00:15:42 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428831/ [7428833] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:25:09 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428833/ [7428834] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} time: 00:16:06 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428834/ [7428835] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} time: 00:26:32 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428835/ [7428852] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_omap_write} time: 00:22:05 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428852/ [7428855] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_fio} time: 00:16:59 info: http://pulpito.ceph.com/yingxin-2023-10-16_01:14:20-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-2-distro-default-smithi/7428855/