2022-10-23T09:51:29.565 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9
2022-10-23T09:51:29.567 INFO:teuthology.results:seed: '1900'
2022-10-23T09:55:16.822 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9
2022-10-23T09:55:16.827 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-10-23T13:32:34.916 INFO:teuthology.results:Tests finished! gathering results...
2022-10-23T13:32:35.653 INFO:scrape:Found 10 jobs
2022-10-23T13:32:36.057 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-618-g255d9465 (255d946547c8b0b68a4803b3e7c6043382f4d014) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7fdf72f9bbee]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x5603eb7360ca]

2022-10-23T13:32:36.059 WARNING:scrape:Ignoring malparsed backtrace: 2022-10-23T10:11:26.859 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738421 got 34359738415 for osd.1
, 2022-10-23T10:11:27.017 INFO:teuthology.orchestra.run.smithi074.stdout:34359738415
, 2022-10-23T10:11:27.033 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738421 got 34359738415 for osd.0

2022-10-23T13:32:36.060 WARNING:scrape:Ignoring malparsed backtrace: 2022-10-23T10:11:34.477 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2022-10-23T10:10:37.438850+0000 time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=10, tm_sec=37, tm_wday=6, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=11, tm_sec=31, tm_wday=6, tm_yday=296, tm_isdst=0)
, 2022-10-23T10:11:34.477 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2022-10-23T10:10:50.273804+0000 time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=10, tm_sec=50, tm_wday=6, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=11, tm_sec=31, tm_wday=6, tm_yday=296, tm_isdst=0)
, 2022-10-23T10:11:34.478 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2022-10-23T10:10:37.438850+0000 time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=10, tm_sec=37, tm_wday=6, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=11, tm_sec=31, tm_wday=6, tm_yday=296, tm_isdst=0)

2022-10-23T13:32:36.061 WARNING:scrape:Ignoring malparsed backtrace: 2022-10-23T10:12:05.594 DEBUG:teuthology.misc:Transferring archived files from smithi080:/var/log/ceph to /home/teuthworker/archive/matan-2022-10-23_09:50:46-crimson-rados-main-distro-crimson-smithi/7078434/remote/smithi080/log
, 2022-10-23T10:12:05.596 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar cz -f - -C /var/log/ceph -- .
, 2022-10-23T10:12:06.135 DEBUG:teuthology.run_tasks:Unwinding manager install

2022-10-23T13:32:36.062 WARNING:scrape:Ignoring malparsed backtrace: 
, 2022-10-23T10:12:27.730 INFO:teuthology.task.internal: dump_program: ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 40000
, 2022-10-23T10:12:27.731 DEBUG:teuthology.orchestra.run.smithi074:> which 'ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 40000'

2022-10-23T13:32:36.085 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-618-g255d9465 (255d946547c8b0b68a4803b3e7c6043382f4d014) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7fdf72f9bbee]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x5603eb7360ca]

2022-10-23T13:32:36.089 WARNING:scrape:Ignoring malparsed backtrace: 2022-10-23T10:11:26.859 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738421 got 34359738415 for osd.1
, 2022-10-23T10:11:27.017 INFO:teuthology.orchestra.run.smithi074.stdout:34359738415
, 2022-10-23T10:11:27.033 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738421 got 34359738415 for osd.0

2022-10-23T13:32:36.090 WARNING:scrape:Ignoring malparsed backtrace: 2022-10-23T10:11:34.477 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2022-10-23T10:10:37.438850+0000 time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=10, tm_sec=37, tm_wday=6, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=11, tm_sec=31, tm_wday=6, tm_yday=296, tm_isdst=0)
, 2022-10-23T10:11:34.477 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2022-10-23T10:10:50.273804+0000 time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=10, tm_sec=50, tm_wday=6, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=11, tm_sec=31, tm_wday=6, tm_yday=296, tm_isdst=0)
, 2022-10-23T10:11:34.478 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2022-10-23T10:10:37.438850+0000 time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=10, tm_sec=37, tm_wday=6, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=11, tm_sec=31, tm_wday=6, tm_yday=296, tm_isdst=0)

2022-10-23T13:32:36.092 WARNING:scrape:Ignoring malparsed backtrace: 2022-10-23T10:12:05.594 DEBUG:teuthology.misc:Transferring archived files from smithi080:/var/log/ceph to /home/teuthworker/archive/matan-2022-10-23_09:50:46-crimson-rados-main-distro-crimson-smithi/7078434/remote/smithi080/log
, 2022-10-23T10:12:05.596 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar cz -f - -C /var/log/ceph -- .
, 2022-10-23T10:12:06.135 DEBUG:teuthology.run_tasks:Unwinding manager install

2022-10-23T13:32:36.093 WARNING:scrape:Ignoring malparsed backtrace: 
, 2022-10-23T10:12:27.730 INFO:teuthology.task.internal: dump_program: ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 40000
, 2022-10-23T10:12:27.731 DEBUG:teuthology.orchestra.run.smithi074:> which 'ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 40000'

2022-10-23T13:32:36.119 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-618-g255d9465 (255d946547c8b0b68a4803b3e7c6043382f4d014) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7fdf72f9bbee]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x5603eb7360ca]

2022-10-23T13:32:36.121 WARNING:scrape:Ignoring malparsed backtrace: 2022-10-23T10:11:26.859 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738421 got 34359738415 for osd.1
, 2022-10-23T10:11:27.017 INFO:teuthology.orchestra.run.smithi074.stdout:34359738415
, 2022-10-23T10:11:27.033 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738421 got 34359738415 for osd.0

2022-10-23T13:32:36.121 WARNING:scrape:Ignoring malparsed backtrace: 2022-10-23T10:11:34.477 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2022-10-23T10:10:37.438850+0000 time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=10, tm_sec=37, tm_wday=6, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=11, tm_sec=31, tm_wday=6, tm_yday=296, tm_isdst=0)
, 2022-10-23T10:11:34.477 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2022-10-23T10:10:50.273804+0000 time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=10, tm_sec=50, tm_wday=6, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=11, tm_sec=31, tm_wday=6, tm_yday=296, tm_isdst=0)
, 2022-10-23T10:11:34.478 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2022-10-23T10:10:37.438850+0000 time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=10, tm_sec=37, tm_wday=6, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=11, tm_sec=31, tm_wday=6, tm_yday=296, tm_isdst=0)

2022-10-23T13:32:36.122 WARNING:scrape:Ignoring malparsed backtrace: 2022-10-23T10:12:05.594 DEBUG:teuthology.misc:Transferring archived files from smithi080:/var/log/ceph to /home/teuthworker/archive/matan-2022-10-23_09:50:46-crimson-rados-main-distro-crimson-smithi/7078434/remote/smithi080/log
, 2022-10-23T10:12:05.596 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar cz -f - -C /var/log/ceph -- .
, 2022-10-23T10:12:06.135 DEBUG:teuthology.run_tasks:Unwinding manager install

2022-10-23T13:32:36.123 WARNING:scrape:Ignoring malparsed backtrace: 
, 2022-10-23T10:12:27.730 INFO:teuthology.task.internal: dump_program: ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 40000
, 2022-10-23T10:12:27.731 DEBUG:teuthology.orchestra.run.smithi074:> which 'ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 40000'

2022-10-23T13:32:36.241 INFO:scrape:Found 5 distinct failure reasons
2022-10-23T13:32:36.242 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd_python.sh
2022-10-23T13:32:36.243 INFO:scrape:1 jobs: ['7078440']
2022-10-23T13:32:36.243 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}']
2022-10-23T13:32:36.244 INFO:scrape:
2022-10-23T13:32:36.244 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd.sh
2022-10-23T13:32:36.244 INFO:scrape:1 jobs: ['7078432']
2022-10-23T13:32:36.245 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}']
2022-10-23T13:32:36.245 INFO:scrape:
2022-10-23T13:32:36.246 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0'
2022-10-23T13:32:36.246 INFO:scrape:1 jobs: ['7078434']
2022-10-23T13:32:36.247 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}']
2022-10-23T13:32:36.247 INFO:scrape:
2022-10-23T13:32:36.247 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_lock_fence.sh
2022-10-23T13:32:36.248 INFO:scrape:1 jobs: ['7078437']
2022-10-23T13:32:36.248 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_lock_and_fence}']
2022-10-23T13:32:36.249 INFO:scrape:
2022-10-23T13:32:36.249 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh
2022-10-23T13:32:36.250 INFO:scrape:1 jobs: ['7078431']
2022-10-23T13:32:36.250 INFO:scrape:suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'deploy/ceph', 'tasks/rados_api_tests}']
2022-10-23T13:32:36.251 INFO:scrape: