2023-10-19T17:15:11.410 INFO:root:teuthology version: 0.0.1.dev201+g202b180
2023-10-19T17:15:11.412 INFO:teuthology.results:subset: '111/120000'
2023-10-19T17:15:11.412 INFO:teuthology.results:seed: '9009'
2023-10-20T04:19:33.752 INFO:root:teuthology version: 0.0.1.dev201+g202b180
2023-10-20T04:19:33.789 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-10-20T16:20:31.250 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2023-10-20T16:20:35.661 INFO:scrape:Found 68 jobs
2023-10-20T16:20:38.023 WARNING:scrape:Misunderstood line: 
2023-10-20T16:20:46.332 WARNING:scrape:Misunderstood line: 
2023-10-20T16:20:53.868 WARNING:scrape:Misunderstood line: 
2023-10-20T16:20:55.637 WARNING:scrape:Misunderstood line: 
2023-10-20T16:20:57.658 WARNING:scrape:Misunderstood line: 
2023-10-20T16:20:57.659 INFO:scrape:Found 13 distinct failure reasons
2023-10-20T16:20:57.659 INFO:scrape:Dead: b'2023-10-20T05:36:28.738 DEBUG:teuthology.exit:Finished running handlers'
2023-10-20T16:20:57.660 INFO:scrape:38 jobs
2023-10-20T16:20:57.660 INFO:scrape:suites intersection: ['rados']
2023-10-20T16:20:57.661 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'cluster/{12-osds', 'clusters/3-mons', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'fast/fast', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/mon-delay', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/monthrash/{ceph', 'rados/singleton-nomsgr/{all/admin_socket_output', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash-erasure-code/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/careful', 'thrashers/default', 'thrashers/fastread', 'thrashers/mapgap', 'thrashers/minsize_recovery', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/one', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-pool-snaps}', 'workloads/cache-snaps}', 'workloads/dedup-io-snaps}', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-rados-plugin=clay-k=4-m=2}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=3-m=1}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/ec-small-objects-fast-read-overwrites}', 'workloads/ec-small-objects-fast-read}', 'workloads/ec-small-objects-overwrites}', 'workloads/ec-snaps-few-objects-overwrites}', 'workloads/pool-snaps-few-objects}', 'workloads/small-objects-balanced}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}']
2023-10-20T16:20:57.661 INFO:scrape:
2023-10-20T16:20:57.662 INFO:scrape:Failure: Command failed on smithi131 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'mkdir $TESTDIR/kvtest && cd $TESTDIR/kvtest && ceph_test_keyvaluedb'"
2023-10-20T16:20:57.662 INFO:scrape:4 jobs: ['7432465', '7432436', '7432433', '7432466']
2023-10-20T16:20:57.663 INFO:scrape:suites intersection: []
2023-10-20T16:20:57.663 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{default}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/objectstore/{backends/keyvaluedb', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-snaps-balanced}']
2023-10-20T16:20:57.664 INFO:scrape:
2023-10-20T16:20:57.664 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55f572d884b6]
2023-10-20T16:20:57.664 INFO:scrape:ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev)
 1: /lib64/libc.so.6(+0x54db0) [0x7f1b8de54db0]
 2: /lib64/libc.so.6(+0xa154c) [0x7f1b8dea154c]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55f572d884b6]
 6: ceph-osd(+0x46e617) [0x55f572d88617]
 7: ceph-osd(+0x42a15e) [0x55f572d4415e]
 8: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x55f57303d492]
 9: ceph-osd(+0x56b441) [0x55f572e85441]
 10: ceph-osd(+0xea9e4b) [0x55f5737c3e4b]
 11: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x55f572fcfec8]
 12: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x1f) [0x55f573109baf]
 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xb97) [0x55f572f1b307]
 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x262) [0x55f5733e50b2]
 15: ceph-osd(+0xacb624) [0x55f5733e5624]
 16: /lib64/libc.so.6(+0x9f802) [0x7f1b8de9f802]
 17: /lib64/libc.so.6(+0x3f450) [0x7f1b8de3f450]
2023-10-20T16:20:57.665 INFO:scrape:1 jobs: ['7432472']
2023-10-20T16:20:57.665 INFO:scrape:suites: ['clusters/{fixed-4', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}']
2023-10-20T16:20:57.666 INFO:scrape:
2023-10-20T16:20:57.666 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55f3a7f034f7]
2023-10-20T16:20:57.666 INFO:scrape:ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f6fb96f2520]
 2: pthread_kill()
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x55f3a7f034f7]
 6: ceph-osd(+0xcd4658) [0x55f3a7f03658]
 7: ceph-osd(+0xc8fd62) [0x55f3a7ebed62]
 8: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x55f3a81524a2]
 9: ceph-osd(+0xd7e9c1) [0x55f3a7fad9c1]
 10: ceph-osd(+0x1695efb) [0x55f3a88c4efb]
 11: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x55f3a80ef3d8]
 12: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x1f) [0x55f3a8224fbf]
 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55f3a8043fa3]
 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x55f3a84f4923]
 15: ceph-osd(+0x12c5e54) [0x55f3a84f4e54]
 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f6fb9744b43]
 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f6fb97d6a00]
2023-10-20T16:20:57.667 INFO:scrape:1 jobs: ['7432474']
2023-10-20T16:20:57.667 INFO:scrape:suites: ['cluster/{12-osds', 'mon_election/classic', 'msgr-failures/osd-dispatch-delay', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}']
2023-10-20T16:20:57.668 INFO:scrape:
2023-10-20T16:20:57.668 INFO:scrape:Valgrind: mon (Leak_StillReachable)
2023-10-20T16:20:57.668 INFO:scrape:5 jobs: ['7432469', '7432488', '7432454', '7432470', '7432441']
2023-10-20T16:20:57.669 INFO:scrape:suites intersection: []
2023-10-20T16:20:57.669 INFO:scrape:suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'validater/valgrind}']
2023-10-20T16:20:57.670 INFO:scrape:
2023-10-20T16:20:57.670 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x56537327c4f7]
2023-10-20T16:20:57.670 INFO:scrape:ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff0c0781520]
 2: pthread_kill()
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x56537327c4f7]
 6: ceph-osd(+0xcd4658) [0x56537327c658]
 7: ceph-osd(+0xc8fd62) [0x565373237d62]
 8: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x5653734cb4a2]
 9: ceph-osd(+0xd7e9c1) [0x5653733269c1]
 10: ceph-osd(+0x1695efb) [0x565373c3defb]
 11: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x5653734683d8]
 12: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x1f) [0x56537359dfbf]
 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5653733bcfa3]
 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x56537386d923]
 15: ceph-osd(+0x12c5e54) [0x56537386de54]
 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff0c07d3b43]
 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff0c0865a00]
2023-10-20T16:20:57.671 INFO:scrape:1 jobs: ['7432476']
2023-10-20T16:20:57.671 INFO:scrape:suites: ['clusters/{fixed-4', 'mon_election/classic', 'msgr-failures/fastclose', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}']
2023-10-20T16:20:57.671 INFO:scrape:
2023-10-20T16:20:57.672 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh
2023-10-20T16:20:57.672 INFO:scrape:2 jobs: ['7432483', '7432449']
2023-10-20T16:20:57.673 INFO:scrape:suites intersection: ['0-start', '1-tasks', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default']
2023-10-20T16:20:57.673 INFO:scrape:suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'rados/upgrade/parallel/{0-random-distro$/{centos_8.stream_container_tools}', 'rados/upgrade/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default']
2023-10-20T16:20:57.673 INFO:scrape:
2023-10-20T16:20:57.674 INFO:scrape:Dead: b'2023-10-20T16:20:45.553 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight'
2023-10-20T16:20:57.674 INFO:scrape:1 jobs: ['7432499']
2023-10-20T16:20:57.675 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'openstack}', 'rados', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_latest}', 'thrashers/minsize_recovery', 'thrashosds-health', 'workloads/ec-snaps-few-objects-overwrites}']
2023-10-20T16:20:57.675 INFO:scrape:
2023-10-20T16:20:57.675 INFO:scrape:Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi133 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8a5df55d22d3638066184c241442ff00fdc0ebdd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh'
2023-10-20T16:20:57.676 INFO:scrape:2 jobs: ['7432461', '7432498']
2023-10-20T16:20:57.676 INFO:scrape:suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'tasks/e2e}']
2023-10-20T16:20:57.676 INFO:scrape:suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-zstd}', 'random-objectstore$/{bluestore-stupid}', 'tasks/e2e}']
2023-10-20T16:20:57.677 INFO:scrape:
2023-10-20T16:20:57.677 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x56518b9624f7]
2023-10-20T16:20:57.678 INFO:scrape:ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fb0a7a05520]
 2: pthread_kill()
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x56518b9624f7]
 6: ceph-osd(+0xcd4658) [0x56518b962658]
 7: ceph-osd(+0xc8fd62) [0x56518b91dd62]
 8: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x56518bbb14a2]
 9: ceph-osd(+0xd7e9c1) [0x56518ba0c9c1]
 10: ceph-osd(+0x1695efb) [0x56518c323efb]
 11: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x56518bb4e3d8]
 12: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x1f) [0x56518bc83fbf]
 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x56518baa2fa3]
 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x56518bf53923]
 15: ceph-osd(+0x12c5e54) [0x56518bf53e54]
 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fb0a7a57b43]
 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fb0a7ae9a00]
2023-10-20T16:20:57.678 INFO:scrape:1 jobs: ['7432434']
2023-10-20T16:20:57.678 INFO:scrape:suites: ['clusters/{fixed-4', 'mon_election/classic', 'msgr-failures/osd-dispatch-delay', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}']
2023-10-20T16:20:57.679 INFO:scrape:
2023-10-20T16:20:57.679 INFO:scrape:Dead: b'2023-10-20T16:20:53.334 INFO:teuthology.orchestra.run.smithi183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused'
2023-10-20T16:20:57.680 INFO:scrape:1 jobs: ['7432496']
2023-10-20T16:20:57.680 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/ec-rados-plugin=isa-k=2-m=1}']
2023-10-20T16:20:57.680 INFO:scrape:
2023-10-20T16:20:57.681 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55de4e6d84b6]
2023-10-20T16:20:57.681 INFO:scrape:ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev)
 1: /lib64/libc.so.6(+0x54db0) [0x7f1a45854db0]
 2: /lib64/libc.so.6(+0xa154c) [0x7f1a458a154c]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55de4e6d84b6]
 6: ceph-osd(+0x46e617) [0x55de4e6d8617]
 7: ceph-osd(+0x42a15e) [0x55de4e69415e]
 8: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x55de4e98d492]
 9: ceph-osd(+0x56b441) [0x55de4e7d5441]
 10: ceph-osd(+0xea9e4b) [0x55de4f113e4b]
 11: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x55de4e91fec8]
 12: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x1f) [0x55de4ea59baf]
 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xb97) [0x55de4e86b307]
 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x262) [0x55de4ed350b2]
 15: ceph-osd(+0xacb624) [0x55de4ed35624]
 16: /lib64/libc.so.6(+0x9f802) [0x7f1a4589f802]
 17: /lib64/libc.so.6(+0x3f450) [0x7f1a4583f450]
2023-10-20T16:20:57.682 INFO:scrape:1 jobs: ['7432460']
2023-10-20T16:20:57.682 INFO:scrape:suites: ['clusters/{fixed-4', 'mon_election/classic', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}']
2023-10-20T16:20:57.682 INFO:scrape:
2023-10-20T16:20:57.683 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x559d1b7194f7]
2023-10-20T16:20:57.683 INFO:scrape:ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f39642c9520]
 2: pthread_kill()
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x559d1b7194f7]
 6: ceph-osd(+0xcd4658) [0x559d1b719658]
 7: ceph-osd(+0xc8fd62) [0x559d1b6d4d62]
 8: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x559d1b9684a2]
 9: ceph-osd(+0xd7e9c1) [0x559d1b7c39c1]
 10: ceph-osd(+0x1695efb) [0x559d1c0daefb]
 11: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x559d1b9053d8]
 12: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x1f) [0x559d1ba3afbf]
 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x559d1b859fa3]
 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x559d1bd0a923]
 15: ceph-osd(+0x12c5e54) [0x559d1bd0ae54]
 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f396431bb43]
 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f39643ada00]
2023-10-20T16:20:57.683 INFO:scrape:1 jobs: ['7432487']
2023-10-20T16:20:57.684 INFO:scrape:suites: ['clusters/{fixed-4', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/thrash-erasure-code-shec/{ceph', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}']
2023-10-20T16:20:57.684 INFO:scrape: