2022-10-23T09:59:42.166 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9
2022-10-23T09:59:42.167 INFO:teuthology.results:seed: '5421'
2022-10-23T10:04:48.237 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9
2022-10-23T10:04:48.243 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-10-23T13:41:06.132 INFO:teuthology.results:Tests finished! gathering results...
2022-10-23T13:41:06.851 INFO:scrape:Found 10 jobs
2022-10-23T13:41:07.120 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-600-gddc8ccb1 (ddc8ccb1557b83c86ae1e5c1ffa02edcfb87046a) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f451f2bdbee]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x55939d5880ca]

2022-10-23T13:41:07.122 WARNING:scrape:Ignoring malparsed backtrace: 2022-10-23T10:20:14.448 INFO:teuthology.orchestra.run.smithi012.stdout:38654705710
, 2022-10-23T10:20:14.462 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705712 got 38654705710 for osd.1
, 2022-10-23T10:20:14.664 INFO:teuthology.orchestra.run.smithi012.stdout:38654705710

2022-10-23T13:41:07.124 WARNING:scrape:Ignoring malparsed backtrace: 2022-10-23T10:20:22.510 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2022-10-23T10:19:42.110965+0000 time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=19, tm_sec=42, tm_wday=6, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=20, tm_sec=19, tm_wday=6, tm_yday=296, tm_isdst=0)
, 2022-10-23T10:20:22.510 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2022-10-23T10:19:28.992160+0000 time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=19, tm_sec=28, tm_wday=6, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=20, tm_sec=19, tm_wday=6, tm_yday=296, tm_isdst=0)
, 2022-10-23T10:20:22.511 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2022-10-23T10:19:42.110965+0000 time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=19, tm_sec=42, tm_wday=6, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=20, tm_sec=19, tm_wday=6, tm_yday=296, tm_isdst=0)

2022-10-23T13:41:07.124 WARNING:scrape:Ignoring malparsed backtrace: 2022-10-23T10:20:58.765 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/log/ceph to /home/teuthworker/archive/matan-2022-10-23_09:53:04-crimson-rados-main-distro-crimson-smithi/7078458/remote/smithi012/log
, 2022-10-23T10:20:58.767 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar cz -f - -C /var/log/ceph -- .
, 2022-10-23T10:20:59.270 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/matan-2022-10-23_09:53:04-crimson-rados-main-distro-crimson-smithi/7078458/remote/smithi053/log

2022-10-23T13:41:07.125 WARNING:scrape:Ignoring malparsed backtrace: 
, 2022-10-23T10:21:18.944 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ...
, 2022-10-23T10:21:20.609 INFO:teuthology.task.internal: core looks like: /tmp/tmpzg25bqne: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 40000', real uid: 1000, effective uid: 1000, real gid: 1271, effective gid: 1271, execfn: '/usr/bin/ceph_test_rados', platform: 'x86_64'

2022-10-23T13:41:07.149 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-600-gddc8ccb1 (ddc8ccb1557b83c86ae1e5c1ffa02edcfb87046a) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f451f2bdbee]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x55939d5880ca]

2022-10-23T13:41:07.151 WARNING:scrape:Ignoring malparsed backtrace: 2022-10-23T10:20:14.448 INFO:teuthology.orchestra.run.smithi012.stdout:38654705710
, 2022-10-23T10:20:14.462 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705712 got 38654705710 for osd.1
, 2022-10-23T10:20:14.664 INFO:teuthology.orchestra.run.smithi012.stdout:38654705710

2022-10-23T13:41:07.152 WARNING:scrape:Ignoring malparsed backtrace: 2022-10-23T10:20:22.510 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2022-10-23T10:19:42.110965+0000 time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=19, tm_sec=42, tm_wday=6, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=20, tm_sec=19, tm_wday=6, tm_yday=296, tm_isdst=0)
, 2022-10-23T10:20:22.510 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2022-10-23T10:19:28.992160+0000 time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=19, tm_sec=28, tm_wday=6, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=20, tm_sec=19, tm_wday=6, tm_yday=296, tm_isdst=0)
, 2022-10-23T10:20:22.511 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2022-10-23T10:19:42.110965+0000 time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=19, tm_sec=42, tm_wday=6, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=20, tm_sec=19, tm_wday=6, tm_yday=296, tm_isdst=0)

2022-10-23T13:41:07.153 WARNING:scrape:Ignoring malparsed backtrace: 2022-10-23T10:20:58.765 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/log/ceph to /home/teuthworker/archive/matan-2022-10-23_09:53:04-crimson-rados-main-distro-crimson-smithi/7078458/remote/smithi012/log
, 2022-10-23T10:20:58.767 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar cz -f - -C /var/log/ceph -- .
, 2022-10-23T10:20:59.270 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/matan-2022-10-23_09:53:04-crimson-rados-main-distro-crimson-smithi/7078458/remote/smithi053/log

2022-10-23T13:41:07.154 WARNING:scrape:Ignoring malparsed backtrace: 
, 2022-10-23T10:21:18.944 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ...
, 2022-10-23T10:21:20.609 INFO:teuthology.task.internal: core looks like: /tmp/tmpzg25bqne: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 40000', real uid: 1000, effective uid: 1000, real gid: 1271, effective gid: 1271, execfn: '/usr/bin/ceph_test_rados', platform: 'x86_64'

2022-10-23T13:41:07.174 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-600-gddc8ccb1 (ddc8ccb1557b83c86ae1e5c1ffa02edcfb87046a) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f451f2bdbee]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x55939d5880ca]

2022-10-23T13:41:07.176 WARNING:scrape:Ignoring malparsed backtrace: 2022-10-23T10:20:14.448 INFO:teuthology.orchestra.run.smithi012.stdout:38654705710
, 2022-10-23T10:20:14.462 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705712 got 38654705710 for osd.1
, 2022-10-23T10:20:14.664 INFO:teuthology.orchestra.run.smithi012.stdout:38654705710

2022-10-23T13:41:07.177 WARNING:scrape:Ignoring malparsed backtrace: 2022-10-23T10:20:22.510 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2022-10-23T10:19:42.110965+0000 time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=19, tm_sec=42, tm_wday=6, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=20, tm_sec=19, tm_wday=6, tm_yday=296, tm_isdst=0)
, 2022-10-23T10:20:22.510 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2022-10-23T10:19:28.992160+0000 time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=19, tm_sec=28, tm_wday=6, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=20, tm_sec=19, tm_wday=6, tm_yday=296, tm_isdst=0)
, 2022-10-23T10:20:22.511 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2022-10-23T10:19:42.110965+0000 time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=19, tm_sec=42, tm_wday=6, tm_yday=296, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=10, tm_mday=23, tm_hour=10, tm_min=20, tm_sec=19, tm_wday=6, tm_yday=296, tm_isdst=0)

2022-10-23T13:41:07.178 WARNING:scrape:Ignoring malparsed backtrace: 2022-10-23T10:20:58.765 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/log/ceph to /home/teuthworker/archive/matan-2022-10-23_09:53:04-crimson-rados-main-distro-crimson-smithi/7078458/remote/smithi012/log
, 2022-10-23T10:20:58.767 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar cz -f - -C /var/log/ceph -- .
, 2022-10-23T10:20:59.270 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/matan-2022-10-23_09:53:04-crimson-rados-main-distro-crimson-smithi/7078458/remote/smithi053/log

2022-10-23T13:41:07.179 WARNING:scrape:Ignoring malparsed backtrace: 
, 2022-10-23T10:21:18.944 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ...
, 2022-10-23T10:21:20.609 INFO:teuthology.task.internal: core looks like: /tmp/tmpzg25bqne: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 40000', real uid: 1000, effective uid: 1000, real gid: 1271, effective gid: 1271, execfn: '/usr/bin/ceph_test_rados', platform: 'x86_64'

2022-10-23T13:41:07.460 INFO:scrape:Found 5 distinct failure reasons
2022-10-23T13:41:07.461 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh
2022-10-23T13:41:07.462 INFO:scrape:1 jobs: ['7078455']
2022-10-23T13:41:07.462 INFO:scrape:suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'deploy/ceph', 'tasks/rados_api_tests}']
2022-10-23T13:41:07.463 INFO:scrape:
2022-10-23T13:41:07.464 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd_python.sh
2022-10-23T13:41:07.465 INFO:scrape:1 jobs: ['7078464']
2022-10-23T13:41:07.465 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}']
2022-10-23T13:41:07.466 INFO:scrape:
2022-10-23T13:41:07.466 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0'
2022-10-23T13:41:07.467 INFO:scrape:1 jobs: ['7078458']
2022-10-23T13:41:07.468 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}']
2022-10-23T13:41:07.468 INFO:scrape:
2022-10-23T13:41:07.469 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd.sh
2022-10-23T13:41:07.469 INFO:scrape:1 jobs: ['7078456']
2022-10-23T13:41:07.470 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}']
2022-10-23T13:41:07.470 INFO:scrape:
2022-10-23T13:41:07.471 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_lock_fence.sh
2022-10-23T13:41:07.471 INFO:scrape:1 jobs: ['7078461']
2022-10-23T13:41:07.471 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_lock_and_fence}']
2022-10-23T13:41:07.472 INFO:scrape: