Found 12 jobs
Ignoring malparsed backtrace:  ceph version 18.0.0-941-ga1b7f9ac (a1b7f9ac6434451a681e5e6e4b41516a163d8b8a) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f3328589bee]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x560c34a140ca]

Ignoring malparsed backtrace: 2022-11-24T08:59:09.930 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2
, 2022-11-24T08:59:09.984 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3
, 2022-11-24T08:59:10.255 INFO:teuthology.orchestra.run.smithi156.stdout:34359740195

Ignoring malparsed backtrace: 2022-11-24T08:59:16.486 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2022-11-24T08:28:45.228524+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=28, tm_sec=45, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=59, tm_sec=13, tm_wday=3, tm_yday=328, tm_isdst=0)
, 2022-11-24T08:59:16.486 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2022-11-24T08:28:45.228524+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=28, tm_sec=45, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=59, tm_sec=13, tm_wday=3, tm_yday=328, tm_isdst=0)
, 2022-11-24T08:59:16.486 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2022-11-24T08:28:52.248816+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=28, tm_sec=52, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=59, tm_sec=13, tm_wday=3, tm_yday=328, tm_isdst=0)

Ignoring malparsed backtrace: 2022-11-24T09:00:07.965 ERROR:teuthology.contextutil:Saw exception from nested tasks
, Traceback (most recent call last):
,   File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 33, in nested

Ignoring malparsed backtrace: /usr/bin/which: unrecognized option '--write-fadvise-dontneed'
, /usr/bin/which: unrecognized option '--max-ops'
, /usr/bin/which: unrecognized option '--objects'

Ignoring malparsed backtrace:  ceph version 18.0.0-941-ga1b7f9ac (a1b7f9ac6434451a681e5e6e4b41516a163d8b8a) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f3328589bee]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x560c34a140ca]

Ignoring malparsed backtrace: 2022-11-24T08:59:09.930 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2
, 2022-11-24T08:59:09.984 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3
, 2022-11-24T08:59:10.255 INFO:teuthology.orchestra.run.smithi156.stdout:34359740195

Ignoring malparsed backtrace: 2022-11-24T08:59:16.486 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2022-11-24T08:28:45.228524+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=28, tm_sec=45, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=59, tm_sec=13, tm_wday=3, tm_yday=328, tm_isdst=0)
, 2022-11-24T08:59:16.486 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2022-11-24T08:28:45.228524+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=28, tm_sec=45, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=59, tm_sec=13, tm_wday=3, tm_yday=328, tm_isdst=0)
, 2022-11-24T08:59:16.486 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2022-11-24T08:28:52.248816+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=28, tm_sec=52, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=59, tm_sec=13, tm_wday=3, tm_yday=328, tm_isdst=0)

Ignoring malparsed backtrace: 2022-11-24T09:00:07.965 ERROR:teuthology.contextutil:Saw exception from nested tasks
, Traceback (most recent call last):
,   File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 33, in nested

Ignoring malparsed backtrace: /usr/bin/which: unrecognized option '--write-fadvise-dontneed'
, /usr/bin/which: unrecognized option '--max-ops'
, /usr/bin/which: unrecognized option '--objects'

Ignoring malparsed backtrace:  ceph version 18.0.0-941-ga1b7f9ac (a1b7f9ac6434451a681e5e6e4b41516a163d8b8a) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f3328589bee]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x560c34a140ca]

Ignoring malparsed backtrace: 2022-11-24T08:59:09.930 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2
, 2022-11-24T08:59:09.984 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3
, 2022-11-24T08:59:10.255 INFO:teuthology.orchestra.run.smithi156.stdout:34359740195

Ignoring malparsed backtrace: 2022-11-24T08:59:16.486 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2022-11-24T08:28:45.228524+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=28, tm_sec=45, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=59, tm_sec=13, tm_wday=3, tm_yday=328, tm_isdst=0)
, 2022-11-24T08:59:16.486 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2022-11-24T08:28:45.228524+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=28, tm_sec=45, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=59, tm_sec=13, tm_wday=3, tm_yday=328, tm_isdst=0)
, 2022-11-24T08:59:16.486 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2022-11-24T08:28:52.248816+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=28, tm_sec=52, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=59, tm_sec=13, tm_wday=3, tm_yday=328, tm_isdst=0)

Ignoring malparsed backtrace: 2022-11-24T09:00:07.965 ERROR:teuthology.contextutil:Saw exception from nested tasks
, Traceback (most recent call last):
,   File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/teuthology/contextutil.py", line 33, in nested

Ignoring malparsed backtrace: /usr/bin/which: unrecognized option '--write-fadvise-dontneed'
, /usr/bin/which: unrecognized option '--max-ops'
, /usr/bin/which: unrecognized option '--objects'

Ignoring malparsed backtrace:  ceph version 18.0.0-941-ga1b7f9ac (a1b7f9ac6434451a681e5e6e4b41516a163d8b8a) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f34f4e06bee]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x555f77e010ca]

Ignoring malparsed backtrace: 2022-11-24T08:36:00.137 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359740201 got 34359740199 for osd.1
, 2022-11-24T08:36:00.156 INFO:teuthology.orchestra.run.smithi167.stdout:38654707494
, 2022-11-24T08:36:00.170 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654707496 got 38654707494 for osd.3

Ignoring malparsed backtrace: 2022-11-24T08:36:07.723 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2022-11-24T08:05:42.342351+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=5, tm_sec=42, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=36, tm_sec=4, tm_wday=3, tm_yday=328, tm_isdst=0)
, 2022-11-24T08:36:07.723 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2022-11-24T08:05:30.549069+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=5, tm_sec=30, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=36, tm_sec=4, tm_wday=3, tm_yday=328, tm_isdst=0)
, 2022-11-24T08:36:07.723 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2022-11-24T08:05:28.540645+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=5, tm_sec=28, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=36, tm_sec=4, tm_wday=3, tm_yday=328, tm_isdst=0)

Ignoring malparsed backtrace: 2022-11-24T08:36:54.052 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2022-11-24_07:12:38-crimson-rados-main-distro-crimson-smithi/7091002/remote/smithi167/crash
, 2022-11-24T08:36:54.053 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar cz -f - -C /var/lib/ceph/crash -- .
, 2022-11-24T08:36:54.115 DEBUG:teuthology.misc:Transferring archived files from smithi201:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2022-11-24_07:12:38-crimson-rados-main-distro-crimson-smithi/7091002/remote/smithi201/crash

Ignoring malparsed backtrace: 2022-11-24T08:38:16.554 DEBUG:teuthology.orchestra.run:got remote process result: 1
, 2022-11-24T08:38:16.555 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive
, 2022-11-24T08:38:16.565 INFO:teuthology.task.internal:Transferring archived files...

Ignoring malparsed backtrace:  ceph version 18.0.0-941-ga1b7f9ac (a1b7f9ac6434451a681e5e6e4b41516a163d8b8a) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f34f4e06bee]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x555f77e010ca]

Ignoring malparsed backtrace: 2022-11-24T08:36:00.137 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359740201 got 34359740199 for osd.1
, 2022-11-24T08:36:00.156 INFO:teuthology.orchestra.run.smithi167.stdout:38654707494
, 2022-11-24T08:36:00.170 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654707496 got 38654707494 for osd.3

Ignoring malparsed backtrace: 2022-11-24T08:36:07.723 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2022-11-24T08:05:42.342351+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=5, tm_sec=42, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=36, tm_sec=4, tm_wday=3, tm_yday=328, tm_isdst=0)
, 2022-11-24T08:36:07.723 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2022-11-24T08:05:30.549069+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=5, tm_sec=30, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=36, tm_sec=4, tm_wday=3, tm_yday=328, tm_isdst=0)
, 2022-11-24T08:36:07.723 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2022-11-24T08:05:28.540645+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=5, tm_sec=28, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=36, tm_sec=4, tm_wday=3, tm_yday=328, tm_isdst=0)

Ignoring malparsed backtrace: 2022-11-24T08:36:54.052 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2022-11-24_07:12:38-crimson-rados-main-distro-crimson-smithi/7091002/remote/smithi167/crash
, 2022-11-24T08:36:54.053 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar cz -f - -C /var/lib/ceph/crash -- .
, 2022-11-24T08:36:54.115 DEBUG:teuthology.misc:Transferring archived files from smithi201:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2022-11-24_07:12:38-crimson-rados-main-distro-crimson-smithi/7091002/remote/smithi201/crash

Ignoring malparsed backtrace: 2022-11-24T08:38:16.554 DEBUG:teuthology.orchestra.run:got remote process result: 1
, 2022-11-24T08:38:16.555 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive
, 2022-11-24T08:38:16.565 INFO:teuthology.task.internal:Transferring archived files...

Ignoring malparsed backtrace:  ceph version 18.0.0-941-ga1b7f9ac (a1b7f9ac6434451a681e5e6e4b41516a163d8b8a) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f34f4e06bee]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x555f77e010ca]

Ignoring malparsed backtrace: 2022-11-24T08:36:00.137 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359740201 got 34359740199 for osd.1
, 2022-11-24T08:36:00.156 INFO:teuthology.orchestra.run.smithi167.stdout:38654707494
, 2022-11-24T08:36:00.170 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654707496 got 38654707494 for osd.3

Ignoring malparsed backtrace: 2022-11-24T08:36:07.723 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2022-11-24T08:05:42.342351+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=5, tm_sec=42, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=36, tm_sec=4, tm_wday=3, tm_yday=328, tm_isdst=0)
, 2022-11-24T08:36:07.723 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2022-11-24T08:05:30.549069+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=5, tm_sec=30, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=36, tm_sec=4, tm_wday=3, tm_yday=328, tm_isdst=0)
, 2022-11-24T08:36:07.723 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2022-11-24T08:05:28.540645+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=5, tm_sec=28, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=36, tm_sec=4, tm_wday=3, tm_yday=328, tm_isdst=0)

Ignoring malparsed backtrace: 2022-11-24T08:36:54.052 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2022-11-24_07:12:38-crimson-rados-main-distro-crimson-smithi/7091002/remote/smithi167/crash
, 2022-11-24T08:36:54.053 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar cz -f - -C /var/lib/ceph/crash -- .
, 2022-11-24T08:36:54.115 DEBUG:teuthology.misc:Transferring archived files from smithi201:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2022-11-24_07:12:38-crimson-rados-main-distro-crimson-smithi/7091002/remote/smithi201/crash

Ignoring malparsed backtrace: 2022-11-24T08:38:16.554 DEBUG:teuthology.orchestra.run:got remote process result: 1
, 2022-11-24T08:38:16.555 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive
, 2022-11-24T08:38:16.565 INFO:teuthology.task.internal:Transferring archived files...

Ignoring malparsed backtrace:  ceph version 18.0.0-941-ga1b7f9ac (a1b7f9ac6434451a681e5e6e4b41516a163d8b8a) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f34f4e06bee]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x555f77e010ca]

Ignoring malparsed backtrace: 2022-11-24T08:36:00.137 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359740201 got 34359740199 for osd.1
, 2022-11-24T08:36:00.156 INFO:teuthology.orchestra.run.smithi167.stdout:38654707494
, 2022-11-24T08:36:00.170 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654707496 got 38654707494 for osd.3

Ignoring malparsed backtrace: 2022-11-24T08:36:07.723 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2022-11-24T08:05:42.342351+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=5, tm_sec=42, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=36, tm_sec=4, tm_wday=3, tm_yday=328, tm_isdst=0)
, 2022-11-24T08:36:07.723 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2022-11-24T08:05:30.549069+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=5, tm_sec=30, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=36, tm_sec=4, tm_wday=3, tm_yday=328, tm_isdst=0)
, 2022-11-24T08:36:07.723 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2022-11-24T08:05:28.540645+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=5, tm_sec=28, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=36, tm_sec=4, tm_wday=3, tm_yday=328, tm_isdst=0)

Ignoring malparsed backtrace: 2022-11-24T08:36:54.052 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2022-11-24_07:12:38-crimson-rados-main-distro-crimson-smithi/7091002/remote/smithi167/crash
, 2022-11-24T08:36:54.053 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar cz -f - -C /var/lib/ceph/crash -- .
, 2022-11-24T08:36:54.115 DEBUG:teuthology.misc:Transferring archived files from smithi201:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2022-11-24_07:12:38-crimson-rados-main-distro-crimson-smithi/7091002/remote/smithi201/crash

Ignoring malparsed backtrace: 2022-11-24T08:38:16.554 DEBUG:teuthology.orchestra.run:got remote process result: 1
, 2022-11-24T08:38:16.555 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive
, 2022-11-24T08:38:16.565 INFO:teuthology.task.internal:Transferring archived files...

Ignoring malparsed backtrace:  ceph version 18.0.0-941-ga1b7f9ac (a1b7f9ac6434451a681e5e6e4b41516a163d8b8a) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f34f4e06bee]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x555f77e010ca]

Ignoring malparsed backtrace: 2022-11-24T08:36:00.137 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359740201 got 34359740199 for osd.1
, 2022-11-24T08:36:00.156 INFO:teuthology.orchestra.run.smithi167.stdout:38654707494
, 2022-11-24T08:36:00.170 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654707496 got 38654707494 for osd.3

Ignoring malparsed backtrace: 2022-11-24T08:36:07.723 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2022-11-24T08:05:42.342351+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=5, tm_sec=42, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=36, tm_sec=4, tm_wday=3, tm_yday=328, tm_isdst=0)
, 2022-11-24T08:36:07.723 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2022-11-24T08:05:30.549069+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=5, tm_sec=30, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=36, tm_sec=4, tm_wday=3, tm_yday=328, tm_isdst=0)
, 2022-11-24T08:36:07.723 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2022-11-24T08:05:28.540645+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=5, tm_sec=28, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=36, tm_sec=4, tm_wday=3, tm_yday=328, tm_isdst=0)

Ignoring malparsed backtrace: 2022-11-24T08:36:54.052 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2022-11-24_07:12:38-crimson-rados-main-distro-crimson-smithi/7091002/remote/smithi167/crash
, 2022-11-24T08:36:54.053 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar cz -f - -C /var/lib/ceph/crash -- .
, 2022-11-24T08:36:54.115 DEBUG:teuthology.misc:Transferring archived files from smithi201:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2022-11-24_07:12:38-crimson-rados-main-distro-crimson-smithi/7091002/remote/smithi201/crash

Ignoring malparsed backtrace: 2022-11-24T08:38:16.554 DEBUG:teuthology.orchestra.run:got remote process result: 1
, 2022-11-24T08:38:16.555 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive
, 2022-11-24T08:38:16.565 INFO:teuthology.task.internal:Transferring archived files...

Ignoring malparsed backtrace:  ceph version 18.0.0-941-ga1b7f9ac (a1b7f9ac6434451a681e5e6e4b41516a163d8b8a) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f34f4e06bee]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x555f77e010ca]

Ignoring malparsed backtrace: 2022-11-24T08:36:00.137 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359740201 got 34359740199 for osd.1
, 2022-11-24T08:36:00.156 INFO:teuthology.orchestra.run.smithi167.stdout:38654707494
, 2022-11-24T08:36:00.170 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654707496 got 38654707494 for osd.3

Ignoring malparsed backtrace: 2022-11-24T08:36:07.723 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2022-11-24T08:05:42.342351+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=5, tm_sec=42, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=36, tm_sec=4, tm_wday=3, tm_yday=328, tm_isdst=0)
, 2022-11-24T08:36:07.723 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2022-11-24T08:05:30.549069+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=5, tm_sec=30, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=36, tm_sec=4, tm_wday=3, tm_yday=328, tm_isdst=0)
, 2022-11-24T08:36:07.723 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2022-11-24T08:05:28.540645+0000 time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=5, tm_sec=28, tm_wday=3, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=36, tm_sec=4, tm_wday=3, tm_yday=328, tm_isdst=0)

Ignoring malparsed backtrace: 2022-11-24T08:36:54.052 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2022-11-24_07:12:38-crimson-rados-main-distro-crimson-smithi/7091002/remote/smithi167/crash
, 2022-11-24T08:36:54.053 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar cz -f - -C /var/lib/ceph/crash -- .
, 2022-11-24T08:36:54.115 DEBUG:teuthology.misc:Transferring archived files from smithi201:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2022-11-24_07:12:38-crimson-rados-main-distro-crimson-smithi/7091002/remote/smithi201/crash

Ignoring malparsed backtrace: 2022-11-24T08:38:16.554 DEBUG:teuthology.orchestra.run:got remote process result: 1
, 2022-11-24T08:38:16.555 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive
, 2022-11-24T08:38:16.565 INFO:teuthology.task.internal:Transferring archived files...

Found 6 distinct failure reasons
Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0'
1 jobs: ['7091005']
suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/write_fadvise_dontneed}']

Failure: Command failed (workunit test rbd/test_librbd_python.sh) on smithi163 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=697b0d99d9a52df92b20c8e053dcd46c75f364a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh'
1 jobs: ['7091004']
suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}']

Timeout 3h running clone.client.0/qa/workunits/rbd/test_lock_fence.sh
1 jobs: ['7091001']
suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_lock_and_fence}']

Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd.sh
1 jobs: ['7090995']
suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}']

Failure: reached maximum tries (500) after waiting for 3000 seconds
1 jobs: ['7091000']
suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/radosbench}']

Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 0 --op snap_remove 0 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0'
1 jobs: ['7091002']
suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}']