2024-03-05T05:47:11.338 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-05T05:47:11.338 INFO:teuthology.results:seed: '1616' 2024-03-05T05:48:19.586 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-05T05:48:19.590 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-05T17:49:13.533 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2024-03-05T17:49:16.281 INFO:scrape:Found 34 jobs 2024-03-05T17:49:16.465 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1794-g61ddc749 (61ddc7499aefcf47a3e5167290880be902d936e2) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fcdd722568f] , 2: ceph_test_rados(+0x1b83a) [0x55d5d00b683a] 2024-03-05T17:49:16.465 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-05T17:49:16.507 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1794-g61ddc749 (61ddc7499aefcf47a3e5167290880be902d936e2) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fcdd722568f] , 2: ceph_test_rados(+0x1b83a) [0x55d5d00b683a] 2024-03-05T17:49:16.507 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-05T17:49:16.541 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1794-g61ddc749 (61ddc7499aefcf47a3e5167290880be902d936e2) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fcdd722568f] , 2: ceph_test_rados(+0x1b83a) [0x55d5d00b683a] 2024-03-05T17:49:16.541 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-05T17:49:16.806 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1794-g61ddc749 (61ddc7499aefcf47a3e5167290880be902d936e2) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f8ccf82568f] , 2: ceph_test_rados(+0x1b3cf) [0x55b54f1873cf] 2024-03-05T17:49:16.807 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-05T17:49:16.887 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1794-g61ddc749 (61ddc7499aefcf47a3e5167290880be902d936e2) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f8ccf82568f] , 2: ceph_test_rados(+0x1b3cf) [0x55b54f1873cf] 2024-03-05T17:49:16.887 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-05T17:49:16.952 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1794-g61ddc749 (61ddc7499aefcf47a3e5167290880be902d936e2) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f8ccf82568f] , 2: ceph_test_rados(+0x1b3cf) [0x55b54f1873cf] 2024-03-05T17:49:16.952 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-05T17:49:17.013 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1794-g61ddc749 (61ddc7499aefcf47a3e5167290880be902d936e2) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f8ccf82568f] , 2: ceph_test_rados(+0x1b3cf) [0x55b54f1873cf] 2024-03-05T17:49:17.014 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-05T17:49:17.615 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1794-g61ddc749 (61ddc7499aefcf47a3e5167290880be902d936e2) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f191ba2568f] , 2: ceph_test_rados(+0x1b83a) [0x55bcbeb2483a] 2024-03-05T17:49:17.616 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-05T17:49:17.758 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1794-g61ddc749 (61ddc7499aefcf47a3e5167290880be902d936e2) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f191ba2568f] , 2: ceph_test_rados(+0x1b83a) [0x55bcbeb2483a] 2024-03-05T17:49:17.758 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-05T17:49:17.935 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1794-g61ddc749 (61ddc7499aefcf47a3e5167290880be902d936e2) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fbe9982568f] , 2: ceph_test_rados(+0x1b83a) [0x55fd3fbbe83a] 2024-03-05T17:49:17.935 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-05T17:49:17.970 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1794-g61ddc749 (61ddc7499aefcf47a3e5167290880be902d936e2) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fbe9982568f] , 2: ceph_test_rados(+0x1b83a) [0x55fd3fbbe83a] 2024-03-05T17:49:17.971 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-05T17:49:18.558 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1794-g61ddc749 (61ddc7499aefcf47a3e5167290880be902d936e2) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f792782568f] , 2: ceph_test_rados(+0x1b3cf) [0x55812dd1d3cf] 2024-03-05T17:49:18.559 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-05T17:49:18.598 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1794-g61ddc749 (61ddc7499aefcf47a3e5167290880be902d936e2) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f792782568f] , 2: ceph_test_rados(+0x1b3cf) [0x55812dd1d3cf] 2024-03-05T17:49:18.598 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-05T17:49:19.219 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1794-g61ddc749 (61ddc7499aefcf47a3e5167290880be902d936e2) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f61c9c2568f] , 2: ceph_test_rados(+0x1b83a) [0x558ef21c383a] 2024-03-05T17:49:19.219 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-05T17:49:19.357 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-1794-g61ddc749 (61ddc7499aefcf47a3e5167290880be902d936e2) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f61c9c2568f] , 2: ceph_test_rados(+0x1b83a) [0x558ef21c383a] 2024-03-05T17:49:19.358 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-05T17:49:19.529 INFO:scrape:Found 7 distinct failure reasons 2024-03-05T17:49:19.530 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2024-03-05T17:49:19.530 INFO:scrape:1 jobs: ['7581857'] 2024-03-05T17:49:19.530 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-localized}'] 2024-03-05T17:49:19.530 INFO:scrape: 2024-03-05T17:49:19.530 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2024-03-05T17:49:19.530 INFO:scrape:5 jobs: ['7581878', '7581869', '7581867', '7581837', '7581875'] 2024-03-05T17:49:19.530 INFO:scrape:suites intersection: ['crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph'] 2024-03-05T17:49:19.530 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/basic/{clusters/fixed-2', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'tasks/readwrite}', 'thrashers/default', 'thrashosds-health', 'workloads/pool-snaps-few-objects}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}'] 2024-03-05T17:49:19.531 INFO:scrape: 2024-03-05T17:49:19.531 INFO:scrape:Failure: reached maximum tries (501) after waiting for 3000 seconds 2024-03-05T17:49:19.531 INFO:scrape:1 jobs: ['7581843'] 2024-03-05T17:49:19.531 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/radosbench-high-concurrency}'] 2024-03-05T17:49:19.531 INFO:scrape: 2024-03-05T17:49:19.531 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2024-03-05T17:49:19.531 INFO:scrape:2 jobs: ['7581830', '7581841'] 2024-03-05T17:49:19.531 INFO:scrape:suites intersection: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph'] 2024-03-05T17:49:19.531 INFO:scrape:suites union: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}', 'tasks/rbd_api_tests}'] 2024-03-05T17:49:19.531 INFO:scrape: 2024-03-05T17:49:19.532 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh --crimson 2024-03-05T17:49:19.532 INFO:scrape:1 jobs: ['7581827'] 2024-03-05T17:49:19.532 INFO:scrape:suites: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}'] 2024-03-05T17:49:19.532 INFO:scrape: 2024-03-05T17:49:19.532 INFO:scrape:Dead: b'2024-03-05T17:47:33.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-05T17:47:33.782+0000 7f5f7506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson' 2024-03-05T17:49:19.532 INFO:scrape:3 jobs: ['7581853', '7581864', '7581883'] 2024-03-05T17:49:19.532 INFO:scrape:suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] 2024-03-05T17:49:19.532 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-balanced}', 'workloads/small-objects}', 'workloads/write_fadvise_dontneed}'] 2024-03-05T17:49:19.532 INFO:scrape: 2024-03-05T17:49:19.532 INFO:scrape:Timeout 1h running clone.client.0/qa/workunits/rados/test_python.sh -m \ 2024-03-05T17:49:19.533 INFO:scrape:1 jobs: ['7581850'] 2024-03-05T17:49:19.533 INFO:scrape:suites: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_python}'] 2024-03-05T17:49:19.533 INFO:scrape: 2024-03-05T17:49:19.534 INFO:teuthology.results:Sending results to yingxin.cheng@intel.com: Test Run: yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/ failed: 11 dead: 0 running: 3 waiting: 0 queued: 0 passed: 20 Fail ================================================================= [7581827] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_api_tests} ----------------------------------------------------------------- time: 03:12:51 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581827/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581827/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e3cc1fa2ad0f44a2b0de9d1c8a96b50c Command failed (workunit test rados/test.sh) on smithi019 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61ddc7499aefcf47a3e5167290880be902d936e2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson' [7581830] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} ----------------------------------------------------------------- time: 03:12:36 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581830/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581830/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cc509246d9864a9abe189e315c4c7bae Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi066 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61ddc7499aefcf47a3e5167290880be902d936e2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crims on_librbd.sh' [7581837] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:22:24 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581837/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581837/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max- ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' [7581841] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} ----------------------------------------------------------------- time: 03:13:51 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581841/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581841/ sentry: https://sentry.ceph.com/organizations/ceph/?query=13398bb7db574109b8ac4ecbe5f86ab9 Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi189 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61ddc7499aefcf47a3e5167290880be902d936e2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' [7581843] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 01:07:22 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581843/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581843/ sentry: https://sentry.ceph.com/organizations/ceph/?query=48b937c61dc449edaabd34b9a4752f8b reached maximum tries (501) after waiting for 3000 seconds [7581850] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_python} ----------------------------------------------------------------- time: 01:12:03 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581850/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581850/ sentry: https://sentry.ceph.com/organizations/ceph/?query=13bc94c04e2c46038355596beaf4456b Command failed (workunit test rados/test_python.sh) on smithi001 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61ddc7499aefcf47a3e5167290880be902d936e2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh -m \'not (wait or tier or ec or bench or stats)\'' [7581857] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:19:53 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581857/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581857/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min- stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' [7581867] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:19:34 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581867/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581867/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' [7581869] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/readwrite} ----------------------------------------------------------------- time: 00:18:03 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581869/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581869/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 45 --op write 22 --op delete 10 --op write_excl 22 --pool unique_pool_0' [7581875] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:20:42 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581875/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581875/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' [7581878] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:23:31 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581878/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581878/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' Running ================================================================= [7581853] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced} info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581853/ [7581864] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581864/ [7581883] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581883/ Pass ================================================================= [7581829] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} time: 00:14:46 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581829/ [7581832] crimson-rados/singleton/{all/osd-backfill crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/bluestore rados} time: 00:15:12 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581832/ [7581834] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:17:22 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581834/ [7581836] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_rw} time: 00:13:59 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581836/ [7581839] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} time: 00:13:32 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581839/ [7581844] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} time: 00:14:45 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581844/ [7581846] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} time: 00:30:19 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581846/ [7581848] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_write} time: 00:15:31 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581848/ [7581852] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} time: 00:12:54 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581852/ [7581855] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read} time: 00:13:15 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581855/ [7581859] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_seq_read} time: 00:13:53 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581859/ [7581860] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} time: 00:11:00 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581860/ [7581862] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} time: 00:13:17 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581862/ [7581866] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} time: 00:12:24 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581866/ [7581871] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests} time: 00:22:12 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581871/ [7581873] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} time: 00:12:31 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581873/ [7581876] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_omap_write} time: 00:27:33 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581876/ [7581880] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format} time: 00:22:02 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581880/ [7581882] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_fio} time: 00:17:00 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581882/ [7581885] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} time: 00:12:15 info: https://pulpito.ceph.com/yingxin-2024-03-05_05:46:48-crimson-rados-ci-crimson-load-obc-distro-default-smithi/7581885/