2024-03-27T01:33:46.921 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-27T01:33:46.922 INFO:teuthology.results:seed: '3112' 2024-03-27T01:57:15.623 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-27T01:57:15.649 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-27T02:30:18.380 INFO:teuthology.results:Tests finished! gathering results... 2024-03-27T02:30:19.420 INFO:scrape:Found 34 jobs 2024-03-27T02:30:19.638 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2463-g4254c927 (4254c92712efa5f9025679283afe440af32d0589) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fc53f64707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-03-27T02:30:19.638 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-27T02:30:19.668 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2463-g4254c927 (4254c92712efa5f9025679283afe440af32d0589) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fc53f64707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-03-27T02:30:19.668 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-27T02:30:19.698 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2463-g4254c927 (4254c92712efa5f9025679283afe440af32d0589) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fc53f64707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-03-27T02:30:19.698 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-27T02:30:19.727 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2463-g4254c927 (4254c92712efa5f9025679283afe440af32d0589) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fc53f64707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-03-27T02:30:19.728 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-27T02:30:20.283 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2463-g4254c927 (4254c92712efa5f9025679283afe440af32d0589) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fa697c4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-03-27T02:30:20.283 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-27T02:30:20.334 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2463-g4254c927 (4254c92712efa5f9025679283afe440af32d0589) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fa697c4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-03-27T02:30:20.334 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-27T02:30:20.379 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2463-g4254c927 (4254c92712efa5f9025679283afe440af32d0589) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fa697c4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-03-27T02:30:20.380 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-27T02:30:20.423 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2463-g4254c927 (4254c92712efa5f9025679283afe440af32d0589) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fa697c4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-03-27T02:30:20.423 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-27T02:30:20.465 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2463-g4254c927 (4254c92712efa5f9025679283afe440af32d0589) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fa697c4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-03-27T02:30:20.465 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-27T02:30:20.510 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2463-g4254c927 (4254c92712efa5f9025679283afe440af32d0589) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fa697c4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-03-27T02:30:20.510 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-27T02:30:20.582 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2463-g4254c927 (4254c92712efa5f9025679283afe440af32d0589) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fcdf1e4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-03-27T02:30:20.583 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-27T02:30:20.619 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2463-g4254c927 (4254c92712efa5f9025679283afe440af32d0589) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fcdf1e4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-03-27T02:30:20.620 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-27T02:30:20.770 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2463-g4254c927 (4254c92712efa5f9025679283afe440af32d0589) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f24cb24707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-03-27T02:30:20.771 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-27T02:30:20.806 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2463-g4254c927 (4254c92712efa5f9025679283afe440af32d0589) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f24cb24707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-03-27T02:30:20.806 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-27T02:30:20.817 INFO:scrape:Found 4 distinct failure reasons 2024-03-27T02:30:20.817 INFO:scrape:Failure: "2024-03-27T02:07:10.861846+0000 mon.a (mon.0) 427 : cluster [ERR] Health check failed: 9 scrub errors (OSD_SCRUB_ERRORS)" in cluster log 2024-03-27T02:30:20.817 INFO:scrape:7 jobs: ['7625029', '7625046', '7625051', '7625045', '7625040', '7625035', '7625021'] 2024-03-27T02:30:20.817 INFO:scrape:suites intersection: ['crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph'] 2024-03-27T02:30:20.817 INFO:scrape:suites union: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}', 'tasks/rbd_api_tests_old_format}', 'tasks/rbd_cls_tests}', 'tasks/rbd_lock_and_fence}', 'tasks/rbd_python_api_tests_old_format}', 'tasks/rbd_python_api_tests}', 'tasks/readwrite}'] 2024-03-27T02:30:20.818 INFO:scrape: 2024-03-27T02:30:20.818 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2024-03-27T02:30:20.818 INFO:scrape:3 jobs: ['7625050', '7625044', '7625027'] 2024-03-27T02:30:20.818 INFO:scrape:suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] 2024-03-27T02:30:20.818 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/pool-snaps-few-objects}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects}'] 2024-03-27T02:30:20.818 INFO:scrape: 2024-03-27T02:30:20.818 INFO:scrape:Failure: Command failed on smithi040 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats' 2024-03-27T02:30:20.819 INFO:scrape:1 jobs: ['7625038'] 2024-03-27T02:30:20.819 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-localized}'] 2024-03-27T02:30:20.819 INFO:scrape: 2024-03-27T02:30:20.819 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2024-03-27T02:30:20.819 INFO:scrape:1 jobs: ['7625042'] 2024-03-27T02:30:20.819 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}'] 2024-03-27T02:30:20.819 INFO:scrape: 2024-03-27T02:30:20.821 INFO:teuthology.results:Sending results to yingxin.cheng@intel.com: Test Run: yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/ failed: 12 dead: 0 running: 0 waiting: 0 queued: 0 passed: 22 Fail ================================================================= [7625021] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:28:17 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625021/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625021/ "2024-03-27T02:11:05.217283+0000 mon.a (mon.0) 1355 : cluster [ERR] Health check failed: 41 scrub errors (OSD_SCRUB_ERRORS)" in cluster log [7625027] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:14:42 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625027/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625027/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max- ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' [7625029] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} ----------------------------------------------------------------- time: 00:19:40 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625029/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625029/ "2024-03-27T02:07:10.861846+0000 mon.a (mon.0) 427 : cluster [ERR] Health check failed: 9 scrub errors (OSD_SCRUB_ERRORS)" in cluster log [7625035] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} ----------------------------------------------------------------- time: 00:12:31 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625035/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625035/ "2024-03-27T02:05:22.461161+0000 mon.a (mon.0) 210 : cluster [ERR] Health check failed: 9 scrub errors (OSD_SCRUB_ERRORS)" in cluster log [7625038] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:25:42 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625038/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625038/ Command failed on smithi040 with status 6: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats' [7625040] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} ----------------------------------------------------------------- time: 00:10:24 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625040/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625040/ "2024-03-27T02:09:51.378169+0000 mon.a (mon.0) 139 : cluster [ERR] Health check failed: 9 scrub errors (OSD_SCRUB_ERRORS)" in cluster log [7625042] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:14:51 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625042/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625042/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' [7625044] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:15:14 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625044/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625044/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' [7625045] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/readwrite} ----------------------------------------------------------------- time: 00:13:07 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625045/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625045/ "2024-03-27T02:14:19.633307+0000 mon.a (mon.0) 167 : cluster [ERR] Health check failed: 9 scrub errors (OSD_SCRUB_ERRORS)" in cluster log [7625046] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests} ----------------------------------------------------------------- time: 00:24:16 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625046/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625046/ "2024-03-27T02:24:54.509045+0000 mon.a (mon.0) 773 : cluster [ERR] Health check failed: 9 scrub errors (OSD_SCRUB_ERRORS)" in cluster log [7625050] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:13:37 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625050/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625050/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' [7625051] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format} ----------------------------------------------------------------- time: 00:20:27 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625051/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625051/ "2024-03-27T02:25:39.807439+0000 mon.a (mon.0) 667 : cluster [ERR] Health check failed: 9 scrub errors (OSD_SCRUB_ERRORS)" in cluster log Pass ================================================================= [7625022] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} time: 00:14:12 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625022/ [7625023] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} time: 00:47:14 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625023/ [7625024] crimson-rados/singleton/{all/osd-backfill crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/bluestore rados} time: 00:14:47 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625024/ [7625025] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:15:04 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625025/ [7625026] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_rw} time: 00:14:13 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625026/ [7625028] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} time: 00:13:42 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625028/ [7625030] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:23:55 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625030/ [7625031] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} time: 00:14:08 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625031/ [7625032] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} time: 00:26:00 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625032/ [7625033] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_write} time: 00:14:27 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625033/ [7625034] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_python} time: 00:13:51 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625034/ [7625036] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:19:42 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625036/ [7625037] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read} time: 00:12:28 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625037/ [7625039] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_seq_read} time: 00:14:10 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625039/ [7625041] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} time: 00:11:47 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625041/ [7625043] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} time: 00:11:14 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625043/ [7625047] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} time: 00:13:12 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625047/ [7625048] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:26:45 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625048/ [7625049] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_omap_write} time: 00:23:13 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625049/ [7625052] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_fio} time: 00:13:59 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625052/ [7625053] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:14:18 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625053/ [7625054] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} time: 00:10:21 info: https://pulpito.ceph.com/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625054/