2024-03-28T05:52:03.849 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-28T05:52:03.849 INFO:teuthology.results:seed: '63' 2024-03-28T12:13:40.469 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-28T12:13:40.517 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-28T12:40:42.487 INFO:teuthology.results:Tests finished! gathering results... 2024-03-28T12:40:43.544 INFO:scrape:Found 34 jobs 2024-03-28T12:40:43.680 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2473-g785fc5e8 (785fc5e8d093374f751c7da3e51e9b56b3718d17) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fc1c424707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-03-28T12:40:43.680 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-28T12:40:43.705 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2473-g785fc5e8 (785fc5e8d093374f751c7da3e51e9b56b3718d17) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fc1c424707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-03-28T12:40:43.705 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-28T12:40:43.730 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2473-g785fc5e8 (785fc5e8d093374f751c7da3e51e9b56b3718d17) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fc1c424707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-03-28T12:40:43.730 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-28T12:40:43.754 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2473-g785fc5e8 (785fc5e8d093374f751c7da3e51e9b56b3718d17) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fc1c424707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-03-28T12:40:43.754 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-28T12:40:43.835 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2473-g785fc5e8 (785fc5e8d093374f751c7da3e51e9b56b3718d17) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fd10044707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-03-28T12:40:43.835 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-28T12:40:43.868 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2473-g785fc5e8 (785fc5e8d093374f751c7da3e51e9b56b3718d17) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fd10044707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] 2024-03-28T12:40:43.869 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2024-03-28T12:40:43.877 INFO:scrape:Found 2 distinct failure reasons 2024-03-28T12:40:43.878 INFO:scrape:Failure: No module named 'tasks.thrashosds' 2024-03-28T12:40:43.878 INFO:scrape:19 jobs: ['7627750', '7627762', '7627748', '7627756', '7627753', '7627755', '7627760', '7627745', '7627758', '7627747', '7627749', '7627743', '7627744', '7627761', '7627763', '7627759', '7627751', '7627757', '7627754'] 2024-03-28T12:40:43.878 INFO:scrape:suites intersection: ['crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph'] 2024-03-28T12:40:43.878 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/basic/{clusters/fixed-2', 'crimson-rados/perf/{clusters/{fixed-2}', 'crimson-rados/rbd/{clusters/fixed-1', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'settings/optimized', 'tasks/rbd_api_tests_old_format}', 'tasks/rbd_cls_tests}', 'tasks/rbd_lock_and_fence}', 'tasks/readwrite}', 'thrashers/default', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/fio_4K_rand_rw}', 'workloads/fio_4M_rand_rw}', 'workloads/fio_4M_rand_write}', 'workloads/pool-snaps-few-objects}', 'workloads/radosbench-high-concurrency}', 'workloads/radosbench_4K_rand_read}', 'workloads/radosbench_4K_seq_read}', 'workloads/radosbench_4M_rand_read}', 'workloads/radosbench_4M_seq_read}', 'workloads/radosbench}', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}', 'workloads/snaps-few-objects-balanced}'] 2024-03-28T12:40:43.878 INFO:scrape: 2024-03-28T12:40:43.878 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2024-03-28T12:40:43.879 INFO:scrape:2 jobs: ['7627766', '7627768'] 2024-03-28T12:40:43.879 INFO:scrape:suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] 2024-03-28T12:40:43.879 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}'] 2024-03-28T12:40:43.879 INFO:scrape: 2024-03-28T12:40:43.880 INFO:teuthology.results:Sending results to yingxin.cheng@intel.com: Test Run: yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/ failed: 21 dead: 0 running: 0 waiting: 0 queued: 0 passed: 13 Fail ================================================================= [7627743] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} ----------------------------------------------------------------- time: 00:09:26 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627743/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627743/ sentry: https://sentry.ceph.com/organizations/ceph/?query=57b28b0bbdd74653bcae9cb0a9c18ff2 No module named 'tasks.ceph' [7627744] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_rw} ----------------------------------------------------------------- time: 00:09:40 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627744/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627744/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9360b5abe4aa440b86a8c7bff8e2925d No module named 'tasks.cbt' [7627745] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:09:06 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627745/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627745/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a266bd24a4a4534a4c4aa9d738ef5e5 No module named 'tasks.ceph' [7627747] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} ----------------------------------------------------------------- time: 00:09:40 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627747/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627747/ sentry: https://sentry.ceph.com/organizations/ceph/?query=64a0ca7c24494c2e9dbbed7d82c5e99b No module named 'tasks.ceph' [7627748] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:10:13 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627748/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627748/ sentry: https://sentry.ceph.com/organizations/ceph/?query=18bce326bf2c425198c881d0a690c6d1 No module named 'tasks.ceph' [7627749] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} ----------------------------------------------------------------- time: 00:09:21 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627749/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627749/ sentry: https://sentry.ceph.com/organizations/ceph/?query=185a3329d43b44c9a86026661bc3f124 No module named 'tasks.ceph' [7627750] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:10:23 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627750/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627750/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9daf66e0f9bc46f99b73f32161107b1f No module named 'tasks.thrashosds' [7627751] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_write} ----------------------------------------------------------------- time: 00:10:08 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627751/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627751/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cbdd3a43e947447e9dc6bd334d6b89da No module named 'tasks.cbt' [7627753] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} ----------------------------------------------------------------- time: 00:09:44 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627753/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627753/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0e5ff12ff8ec4190af930177815519e4 No module named 'tasks.ceph' [7627754] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:09:39 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627754/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627754/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cf946120e44f4195996e6bdf49391b74 No module named 'tasks.ceph' [7627755] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read} ----------------------------------------------------------------- time: 00:09:05 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627755/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627755/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cfe79dd348364f28a92b658121b1b130 No module named 'tasks.ceph' [7627756] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized} ----------------------------------------------------------------- time: 00:09:13 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627756/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627756/ sentry: https://sentry.ceph.com/organizations/ceph/?query=590158e0c9d74479bd9916501e85bbaa No module named 'tasks.ceph' [7627757] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_seq_read} ----------------------------------------------------------------- time: 00:08:49 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627757/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627757/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5bfe5f120ec14fc0b14054d68b3799ae No module named 'tasks.ceph' [7627758] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} ----------------------------------------------------------------- time: 00:09:27 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627758/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627758/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5374ccc64b2b4381abd31ec5a9062b2a No module named 'tasks.workunit' [7627759] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} ----------------------------------------------------------------- time: 00:09:31 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627759/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627759/ sentry: https://sentry.ceph.com/organizations/ceph/?query=426a3bbf157743a0ba674aa15a6fec57 No module named 'tasks.cbt' [7627760] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:08:58 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627760/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627760/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c7c7e112322941108f5fc934500c0371 No module named 'tasks.ceph' [7627761] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} ----------------------------------------------------------------- time: 00:09:27 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627761/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627761/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d8fccdb87f5544cb87f969bb99a0dcb9 No module named 'tasks.ceph' [7627762] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:07:33 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627762/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627762/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8e1dc61e29804ab98eeef17f8d57702b No module named 'tasks' [7627763] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/readwrite} ----------------------------------------------------------------- time: 00:07:47 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627763/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627763/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3195e36a9bce4bdb8355082220ff437b No module named 'tasks' [7627766] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-localized} ----------------------------------------------------------------- time: 00:13:38 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627766/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627766/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride- size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' [7627768] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:15:19 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627768/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627768/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' Pass ================================================================= [7627739] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_api_tests} time: 00:27:59 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627739/ [7627740] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} time: 00:14:03 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627740/ [7627741] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} time: 00:50:29 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627741/ [7627742] crimson-rados/singleton/{all/osd-backfill crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/bluestore rados} time: 00:14:23 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627742/ [7627746] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} time: 00:13:53 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627746/ [7627752] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_python} time: 00:15:26 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627752/ [7627764] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests} time: 00:24:50 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627764/ [7627765] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} time: 00:10:04 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627765/ [7627767] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_omap_write} time: 00:22:34 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627767/ [7627769] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format} time: 00:20:35 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627769/ [7627770] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_fio} time: 00:13:30 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627770/ [7627771] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:13:54 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627771/ [7627772] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} time: 00:10:04 info: https://pulpito.ceph.com/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627772/