Found 9 jobs
Ignoring malparsed backtrace:  ceph version 17.0.0-14583-g281beabd (281beabdfd6958897b81311e92181abac8827ef6) quincy (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f2a297d6cb0]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x588) [0x55db039f83aa]

Ignoring malparsed backtrace: 2022-08-25T14:48:20.677 INFO:teuthology.orchestra.run.smithi040.stdout:38654705690
, 2022-08-25T14:48:20.694 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705695 got 38654705690 for osd.1
, 2022-08-25T14:48:20.696 INFO:teuthology.orchestra.run.smithi040.stdout:38654705690

Ignoring malparsed backtrace: 2022-08-25T14:48:28.597 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2022-08-25T14:48:00.122363+0000 time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=48, tm_sec=0, tm_wday=3, tm_yday=237, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=48, tm_sec=25, tm_wday=3, tm_yday=237, tm_isdst=0)
, 2022-08-25T14:48:28.598 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2022-08-25T14:47:53.096723+0000 time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=47, tm_sec=53, tm_wday=3, tm_yday=237, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=48, tm_sec=25, tm_wday=3, tm_yday=237, tm_isdst=0)
, 2022-08-25T14:48:28.599 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2022-08-25T14:48:00.122363+0000 time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=48, tm_sec=0, tm_wday=3, tm_yday=237, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=48, tm_sec=25, tm_wday=3, tm_yday=237, tm_isdst=0)

Ignoring malparsed backtrace: 2022-08-25T14:48:57.820 INFO:tasks.ceph:Archiving logs...
, 2022-08-25T14:48:57.821 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/rzarzynski-2022-08-25_14:28:41-crimson-rados-main-distro-default-smithi/6992389/remote/smithi040/log
, 2022-08-25T14:48:57.828 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar cz -f - -C /var/log/ceph -- .

Ignoring malparsed backtrace: 2022-08-25T14:49:23.234 INFO:teuthology.task.internal:Transferring binaries for coredumps...
, 2022-08-25T14:49:23.249 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/rzarzynski-2022-08-25_14:28:41-crimson-rados-main-distro-default-smithi/6992389/remote/smithi040/coredump/1661438887.100960.core.gz: gzip compressed data, was "1661438887.100960.core", last modified: Thu Aug 25 14:49:00 2022, max compression
, 

Ignoring malparsed backtrace:  ceph version 17.0.0-14583-g281beabd (281beabdfd6958897b81311e92181abac8827ef6) quincy (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f2a297d6cb0]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x588) [0x55db039f83aa]

Ignoring malparsed backtrace: 2022-08-25T14:48:20.677 INFO:teuthology.orchestra.run.smithi040.stdout:38654705690
, 2022-08-25T14:48:20.694 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705695 got 38654705690 for osd.1
, 2022-08-25T14:48:20.696 INFO:teuthology.orchestra.run.smithi040.stdout:38654705690

Ignoring malparsed backtrace: 2022-08-25T14:48:28.597 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2022-08-25T14:48:00.122363+0000 time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=48, tm_sec=0, tm_wday=3, tm_yday=237, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=48, tm_sec=25, tm_wday=3, tm_yday=237, tm_isdst=0)
, 2022-08-25T14:48:28.598 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2022-08-25T14:47:53.096723+0000 time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=47, tm_sec=53, tm_wday=3, tm_yday=237, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=48, tm_sec=25, tm_wday=3, tm_yday=237, tm_isdst=0)
, 2022-08-25T14:48:28.599 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2022-08-25T14:48:00.122363+0000 time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=48, tm_sec=0, tm_wday=3, tm_yday=237, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=48, tm_sec=25, tm_wday=3, tm_yday=237, tm_isdst=0)

Ignoring malparsed backtrace: 2022-08-25T14:48:57.820 INFO:tasks.ceph:Archiving logs...
, 2022-08-25T14:48:57.821 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/rzarzynski-2022-08-25_14:28:41-crimson-rados-main-distro-default-smithi/6992389/remote/smithi040/log
, 2022-08-25T14:48:57.828 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar cz -f - -C /var/log/ceph -- .

Ignoring malparsed backtrace: 2022-08-25T14:49:23.234 INFO:teuthology.task.internal:Transferring binaries for coredumps...
, 2022-08-25T14:49:23.249 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/rzarzynski-2022-08-25_14:28:41-crimson-rados-main-distro-default-smithi/6992389/remote/smithi040/coredump/1661438887.100960.core.gz: gzip compressed data, was "1661438887.100960.core", last modified: Thu Aug 25 14:49:00 2022, max compression
, 

Ignoring malparsed backtrace:  ceph version 17.0.0-14583-g281beabd (281beabdfd6958897b81311e92181abac8827ef6) quincy (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f2a297d6cb0]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x588) [0x55db039f83aa]

Ignoring malparsed backtrace: 2022-08-25T14:48:20.677 INFO:teuthology.orchestra.run.smithi040.stdout:38654705690
, 2022-08-25T14:48:20.694 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705695 got 38654705690 for osd.1
, 2022-08-25T14:48:20.696 INFO:teuthology.orchestra.run.smithi040.stdout:38654705690

Ignoring malparsed backtrace: 2022-08-25T14:48:28.597 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2022-08-25T14:48:00.122363+0000 time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=48, tm_sec=0, tm_wday=3, tm_yday=237, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=48, tm_sec=25, tm_wday=3, tm_yday=237, tm_isdst=0)
, 2022-08-25T14:48:28.598 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2022-08-25T14:47:53.096723+0000 time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=47, tm_sec=53, tm_wday=3, tm_yday=237, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=48, tm_sec=25, tm_wday=3, tm_yday=237, tm_isdst=0)
, 2022-08-25T14:48:28.599 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2022-08-25T14:48:00.122363+0000 time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=48, tm_sec=0, tm_wday=3, tm_yday=237, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=48, tm_sec=25, tm_wday=3, tm_yday=237, tm_isdst=0)

Ignoring malparsed backtrace: 2022-08-25T14:48:57.820 INFO:tasks.ceph:Archiving logs...
, 2022-08-25T14:48:57.821 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/rzarzynski-2022-08-25_14:28:41-crimson-rados-main-distro-default-smithi/6992389/remote/smithi040/log
, 2022-08-25T14:48:57.828 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar cz -f - -C /var/log/ceph -- .

Ignoring malparsed backtrace: 2022-08-25T14:49:23.234 INFO:teuthology.task.internal:Transferring binaries for coredumps...
, 2022-08-25T14:49:23.249 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/rzarzynski-2022-08-25_14:28:41-crimson-rados-main-distro-default-smithi/6992389/remote/smithi040/coredump/1661438887.100960.core.gz: gzip compressed data, was "1661438887.100960.core", last modified: Thu Aug 25 14:49:00 2022, max compression
, 

Ignoring malparsed backtrace:  ceph version 17.0.0-14583-g281beabd (281beabdfd6958897b81311e92181abac8827ef6) quincy (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f2a297d6cb0]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x588) [0x55db039f83aa]

Ignoring malparsed backtrace: 2022-08-25T14:48:20.677 INFO:teuthology.orchestra.run.smithi040.stdout:38654705690
, 2022-08-25T14:48:20.694 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705695 got 38654705690 for osd.1
, 2022-08-25T14:48:20.696 INFO:teuthology.orchestra.run.smithi040.stdout:38654705690

Ignoring malparsed backtrace: 2022-08-25T14:48:28.597 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2022-08-25T14:48:00.122363+0000 time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=48, tm_sec=0, tm_wday=3, tm_yday=237, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=48, tm_sec=25, tm_wday=3, tm_yday=237, tm_isdst=0)
, 2022-08-25T14:48:28.598 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2022-08-25T14:47:53.096723+0000 time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=47, tm_sec=53, tm_wday=3, tm_yday=237, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=48, tm_sec=25, tm_wday=3, tm_yday=237, tm_isdst=0)
, 2022-08-25T14:48:28.599 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2022-08-25T14:48:00.122363+0000 time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=48, tm_sec=0, tm_wday=3, tm_yday=237, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=8, tm_mday=25, tm_hour=14, tm_min=48, tm_sec=25, tm_wday=3, tm_yday=237, tm_isdst=0)

Ignoring malparsed backtrace: 2022-08-25T14:48:57.820 INFO:tasks.ceph:Archiving logs...
, 2022-08-25T14:48:57.821 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/rzarzynski-2022-08-25_14:28:41-crimson-rados-main-distro-default-smithi/6992389/remote/smithi040/log
, 2022-08-25T14:48:57.828 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar cz -f - -C /var/log/ceph -- .

Ignoring malparsed backtrace: 2022-08-25T14:49:23.234 INFO:teuthology.task.internal:Transferring binaries for coredumps...
, 2022-08-25T14:49:23.249 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/rzarzynski-2022-08-25_14:28:41-crimson-rados-main-distro-default-smithi/6992389/remote/smithi040/coredump/1661438887.100960.core.gz: gzip compressed data, was "1661438887.100960.core", last modified: Thu Aug 25 14:49:00 2022, max compression
, 

Found 4 distinct failure reasons
Timeout 3h running clone.client.0/qa/workunits/rbd/test_lock_fence.sh
1 jobs: ['6992394']
suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_lock_and_fence}']

Failure: expected string or bytes-like object
1 jobs: ['6992386']
suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'deploy/ceph', 'tasks/rados_api_tests}']

Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0'
1 jobs: ['6992389']
suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}']

Failure: Command failed on smithi142 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1'
1 jobs: ['6992387']
suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_api_tests}']