2022-12-07T15:30:09.169 INFO:root:teuthology version: 0.0.1.dev3+g4da97cf
2022-12-07T15:30:09.171 INFO:teuthology.results:seed: '8435'
2022-12-07T15:30:21.557 INFO:root:teuthology version: 0.0.1.dev3+g4da97cf
2022-12-07T15:30:21.558 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-12-07T18:52:38.077 INFO:teuthology.results:Tests finished! gathering results...
2022-12-07T18:52:39.049 INFO:scrape:Found 12 jobs
2022-12-07T18:52:40.948 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-1140-g33572233 (335722333f37d86ea74355c5f88c9a23b08fe7b5) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f1987049bee]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x5651be0f10ca]

2022-12-07T18:52:40.951 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-07T16:20:45.330 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359740197 got 34359740194 for osd.1
, 2022-12-07T16:20:46.265 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3
, 2022-12-07T16:20:46.293 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0

2022-12-07T18:52:40.953 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-07T16:20:53.004 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2022-12-07T15:50:19.609105+0000 time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=15, tm_min=50, tm_sec=19, tm_wday=2, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=16, tm_min=20, tm_sec=49, tm_wday=2, tm_yday=341, tm_isdst=0)
, 2022-12-07T16:20:53.005 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2022-12-07T15:50:26.627975+0000 time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=15, tm_min=50, tm_sec=26, tm_wday=2, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=16, tm_min=20, tm_sec=49, tm_wday=2, tm_yday=341, tm_isdst=0)
, 2022-12-07T16:20:53.005 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2022-12-07T15:50:26.627975+0000 time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=15, tm_min=50, tm_sec=26, tm_wday=2, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=16, tm_min=20, tm_sec=49, tm_wday=2, tm_yday=341, tm_isdst=0)

2022-12-07T18:52:40.953 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-07T16:21:39.485 DEBUG:teuthology.orchestra.run.smithi085:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --
, 2022-12-07T16:21:40.814 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored
, 2022-12-07T16:22:01.253 INFO:tasks.ceph:Archiving logs...

2022-12-07T18:52:40.954 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-07T16:22:41.398 INFO:teuthology.task.internal:Transferring binaries for coredumps...
, 2022-12-07T16:22:41.449 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/matan-2022-12-07_15:30:01-crimson-rados-main-distro-crimson-smithi/7106071/remote/smithi061/coredump/1670430029.105580.core.gz: gzip compressed data, was "1670430029.105580.core", last modified: Wed Dec  7 16:22:12 2022, max compression
, 

2022-12-07T18:52:41.095 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-1140-g33572233 (335722333f37d86ea74355c5f88c9a23b08fe7b5) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f1987049bee]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x5651be0f10ca]

2022-12-07T18:52:41.097 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-07T16:20:45.330 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359740197 got 34359740194 for osd.1
, 2022-12-07T16:20:46.265 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3
, 2022-12-07T16:20:46.293 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0

2022-12-07T18:52:41.098 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-07T16:20:53.004 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2022-12-07T15:50:19.609105+0000 time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=15, tm_min=50, tm_sec=19, tm_wday=2, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=16, tm_min=20, tm_sec=49, tm_wday=2, tm_yday=341, tm_isdst=0)
, 2022-12-07T16:20:53.005 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2022-12-07T15:50:26.627975+0000 time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=15, tm_min=50, tm_sec=26, tm_wday=2, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=16, tm_min=20, tm_sec=49, tm_wday=2, tm_yday=341, tm_isdst=0)
, 2022-12-07T16:20:53.005 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2022-12-07T15:50:26.627975+0000 time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=15, tm_min=50, tm_sec=26, tm_wday=2, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=16, tm_min=20, tm_sec=49, tm_wday=2, tm_yday=341, tm_isdst=0)

2022-12-07T18:52:41.099 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-07T16:21:39.485 DEBUG:teuthology.orchestra.run.smithi085:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --
, 2022-12-07T16:21:40.814 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored
, 2022-12-07T16:22:01.253 INFO:tasks.ceph:Archiving logs...

2022-12-07T18:52:41.099 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-07T16:22:41.398 INFO:teuthology.task.internal:Transferring binaries for coredumps...
, 2022-12-07T16:22:41.449 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/matan-2022-12-07_15:30:01-crimson-rados-main-distro-crimson-smithi/7106071/remote/smithi061/coredump/1670430029.105580.core.gz: gzip compressed data, was "1670430029.105580.core", last modified: Wed Dec  7 16:22:12 2022, max compression
, 

2022-12-07T18:52:41.224 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-1140-g33572233 (335722333f37d86ea74355c5f88c9a23b08fe7b5) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f1987049bee]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x5651be0f10ca]

2022-12-07T18:52:41.227 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-07T16:20:45.330 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359740197 got 34359740194 for osd.1
, 2022-12-07T16:20:46.265 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3
, 2022-12-07T16:20:46.293 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0

2022-12-07T18:52:41.227 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-07T16:20:53.004 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2022-12-07T15:50:19.609105+0000 time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=15, tm_min=50, tm_sec=19, tm_wday=2, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=16, tm_min=20, tm_sec=49, tm_wday=2, tm_yday=341, tm_isdst=0)
, 2022-12-07T16:20:53.005 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2022-12-07T15:50:26.627975+0000 time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=15, tm_min=50, tm_sec=26, tm_wday=2, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=16, tm_min=20, tm_sec=49, tm_wday=2, tm_yday=341, tm_isdst=0)
, 2022-12-07T16:20:53.005 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2022-12-07T15:50:26.627975+0000 time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=15, tm_min=50, tm_sec=26, tm_wday=2, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=16, tm_min=20, tm_sec=49, tm_wday=2, tm_yday=341, tm_isdst=0)

2022-12-07T18:52:41.228 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-07T16:21:39.485 DEBUG:teuthology.orchestra.run.smithi085:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --
, 2022-12-07T16:21:40.814 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored
, 2022-12-07T16:22:01.253 INFO:tasks.ceph:Archiving logs...

2022-12-07T18:52:41.229 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-07T16:22:41.398 INFO:teuthology.task.internal:Transferring binaries for coredumps...
, 2022-12-07T16:22:41.449 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/matan-2022-12-07_15:30:01-crimson-rados-main-distro-crimson-smithi/7106071/remote/smithi061/coredump/1670430029.105580.core.gz: gzip compressed data, was "1670430029.105580.core", last modified: Wed Dec  7 16:22:12 2022, max compression
, 

2022-12-07T18:52:41.352 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-1140-g33572233 (335722333f37d86ea74355c5f88c9a23b08fe7b5) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f1987049bee]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x5651be0f10ca]

2022-12-07T18:52:41.354 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-07T16:20:45.330 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359740197 got 34359740194 for osd.1
, 2022-12-07T16:20:46.265 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3
, 2022-12-07T16:20:46.293 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0

2022-12-07T18:52:41.355 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-07T16:20:53.004 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2022-12-07T15:50:19.609105+0000 time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=15, tm_min=50, tm_sec=19, tm_wday=2, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=16, tm_min=20, tm_sec=49, tm_wday=2, tm_yday=341, tm_isdst=0)
, 2022-12-07T16:20:53.005 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2022-12-07T15:50:26.627975+0000 time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=15, tm_min=50, tm_sec=26, tm_wday=2, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=16, tm_min=20, tm_sec=49, tm_wday=2, tm_yday=341, tm_isdst=0)
, 2022-12-07T16:20:53.005 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2022-12-07T15:50:26.627975+0000 time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=15, tm_min=50, tm_sec=26, tm_wday=2, tm_yday=341, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=12, tm_mday=7, tm_hour=16, tm_min=20, tm_sec=49, tm_wday=2, tm_yday=341, tm_isdst=0)

2022-12-07T18:52:41.356 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-07T16:21:39.485 DEBUG:teuthology.orchestra.run.smithi085:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --
, 2022-12-07T16:21:40.814 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored
, 2022-12-07T16:22:01.253 INFO:tasks.ceph:Archiving logs...

2022-12-07T18:52:41.356 WARNING:scrape:Ignoring malparsed backtrace: 2022-12-07T16:22:41.398 INFO:teuthology.task.internal:Transferring binaries for coredumps...
, 2022-12-07T16:22:41.449 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/matan-2022-12-07_15:30:01-crimson-rados-main-distro-crimson-smithi/7106071/remote/smithi061/coredump/1670430029.105580.core.gz: gzip compressed data, was "1670430029.105580.core", last modified: Wed Dec  7 16:22:12 2022, max compression
, 

2022-12-07T18:52:41.544 INFO:scrape:Found 5 distinct failure reasons
2022-12-07T18:52:41.546 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_lock_fence.sh
2022-12-07T18:52:41.546 INFO:scrape:1 jobs: ['7106070']
2022-12-07T18:52:41.547 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_lock_and_fence}']
2022-12-07T18:52:41.548 INFO:scrape:
2022-12-07T18:52:41.549 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd.sh
2022-12-07T18:52:41.550 INFO:scrape:1 jobs: ['7106064']
2022-12-07T18:52:41.550 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}']
2022-12-07T18:52:41.551 INFO:scrape:
2022-12-07T18:52:41.552 INFO:scrape:Failure: reached maximum tries (500) after waiting for 3000 seconds
2022-12-07T18:52:41.552 INFO:scrape:1 jobs: ['7106069']
2022-12-07T18:52:41.553 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/radosbench}']
2022-12-07T18:52:41.554 INFO:scrape:
2022-12-07T18:52:41.554 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 0 --op snap_remove 0 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0'
2022-12-07T18:52:41.555 INFO:scrape:1 jobs: ['7106071']
2022-12-07T18:52:41.555 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}']
2022-12-07T18:52:41.556 INFO:scrape:
2022-12-07T18:52:41.556 INFO:scrape:Failure: Command failed (workunit test rbd/test_librbd_python.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=697b0d99d9a52df92b20c8e053dcd46c75f364a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh'
2022-12-07T18:52:41.557 INFO:scrape:1 jobs: ['7106073']
2022-12-07T18:52:41.557 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}']
2022-12-07T18:52:41.558 INFO:scrape: