2023-10-17T06:51:25.016 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-17T06:51:25.017 INFO:teuthology.results:seed: '3210' 2023-10-17T07:37:41.861 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-17T07:37:41.865 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-17T19:38:36.538 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-10-17T19:38:38.435 INFO:scrape:Found 25 jobs 2023-10-17T19:38:39.725 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-6790-ga83d4841 (a83d484198747052d1ed9eeed0a72e20b007e411) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fd4fe00ebef] , 2: ceph_test_rados(+0x1d2b3) [0x562e922a12b3] 2023-10-17T19:38:39.726 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2023-10-17T19:38:39.765 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-6790-ga83d4841 (a83d484198747052d1ed9eeed0a72e20b007e411) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fd4fe00ebef] , 2: ceph_test_rados(+0x1d2b3) [0x562e922a12b3] 2023-10-17T19:38:39.766 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2023-10-17T19:38:39.802 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-6790-ga83d4841 (a83d484198747052d1ed9eeed0a72e20b007e411) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fd4fe00ebef] , 2: ceph_test_rados(+0x1d2b3) [0x562e922a12b3] 2023-10-17T19:38:39.803 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2023-10-17T19:38:39.834 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-6790-ga83d4841 (a83d484198747052d1ed9eeed0a72e20b007e411) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fd4fe00ebef] , 2: ceph_test_rados(+0x1d2b3) [0x562e922a12b3] 2023-10-17T19:38:39.835 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2023-10-17T19:38:40.013 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-6790-ga83d4841 (a83d484198747052d1ed9eeed0a72e20b007e411) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fbb9c00ebef] , 2: ceph_test_rados(+0x1d2b3) [0x5632c61df2b3] 2023-10-17T19:38:40.013 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2023-10-17T19:38:40.078 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 18.0.0-6790-ga83d4841 (a83d484198747052d1ed9eeed0a72e20b007e411) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fbb9c00ebef] , 2: ceph_test_rados(+0x1d2b3) [0x5632c61df2b3] 2023-10-17T19:38:40.079 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace! 2023-10-17T19:38:40.208 INFO:scrape:Found 6 distinct failure reasons 2023-10-17T19:38:40.209 INFO:scrape:Failure: "2023-10-17T07:17:50.565784+0000 mon.a (mon.0) 132 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive (PG_AVAILABILITY)" in cluster log 2023-10-17T19:38:40.209 INFO:scrape:7 jobs: ['7431072', '7431087', '7431075', '7431083', '7431089', '7431081', '7431085'] 2023-10-17T19:38:40.210 INFO:scrape:suites intersection: ['crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph'] 2023-10-17T19:38:40.210 INFO:scrape:suites union: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-rados/perf/{clusters/{fixed-2}', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'settings/optimized', 'tasks/readwrite}', 'workloads/fio_4M_rand_read}', 'workloads/fio_4M_rand_rw}', 'workloads/radosbench_4K_rand_read}', 'workloads/radosbench_4K_seq_read}', 'workloads/radosbench_4M_rand_read}', 'workloads/radosbench_4M_seq_read}'] 2023-10-17T19:38:40.211 INFO:scrape: 2023-10-17T19:38:40.211 INFO:scrape:Dead: b'2023-10-17T19:02:29.085 DEBUG:teuthology.exit:Finished running handlers' 2023-10-17T19:38:40.211 INFO:scrape:1 jobs: ['7431071'] 2023-10-17T19:38:40.212 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/pool-snaps-few-objects}'] 2023-10-17T19:38:40.212 INFO:scrape: 2023-10-17T19:38:40.213 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2023-10-17T19:38:40.213 INFO:scrape:2 jobs: ['7431073', '7431067'] 2023-10-17T19:38:40.213 INFO:scrape:suites intersection: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph'] 2023-10-17T19:38:40.214 INFO:scrape:suites union: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}', 'tasks/rbd_api_tests}'] 2023-10-17T19:38:40.214 INFO:scrape: 2023-10-17T19:38:40.215 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_lock.sh 2023-10-17T19:38:40.215 INFO:scrape:1 jobs: ['7431079'] 2023-10-17T19:38:40.216 INFO:scrape:suites: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_cls_tests}'] 2023-10-17T19:38:40.216 INFO:scrape: 2023-10-17T19:38:40.216 INFO:scrape:Dead: b'2023-10-17T19:38:38.495 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-17T19:38:38.494+0000 7f8abbd15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson' 2023-10-17T19:38:40.217 INFO:scrape:2 jobs: ['7431086', '7431088'] 2023-10-17T19:38:40.217 INFO:scrape:suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] 2023-10-17T19:38:40.218 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}', 'workloads/snaps-few-objects-balanced}'] 2023-10-17T19:38:40.218 INFO:scrape: 2023-10-17T19:38:40.219 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2023-10-17T19:38:40.219 INFO:scrape:2 jobs: ['7431082', '7431080'] 2023-10-17T19:38:40.219 INFO:scrape:suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] 2023-10-17T19:38:40.220 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}'] 2023-10-17T19:38:40.220 INFO:scrape: